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FOREWORD 


The  Army  Mathematics  Steering  Committee  (AMSC)  sponsors,  on  behalf  of 
the  Chief  of  Research,  Development  and  Acquisition,  the  series  of  conferences 
entitled,  "Design  of  Experiments  in  Army  Research,  Development  and  Testing." 

It  delegates  the  responsibility  for  the  conduction  of  these  meetings  to  its 
Subcommittee  on  Probability  and  Statistics,  At  the  30  November  1973  meeting 
of  this  subcommittee  it  was  recommended  that  appropriate  steps  be  taken  to 
celebrate  the  twentieth  anniversary  of  the  design  of  experiments  conferences. 
After  some  deliberation  it  was  decided  to  ask  members  of  the  Program 
Committee  to  increase  the  usual  number  of  invited  speakers  from  five  to  eight, 
and  to  invite  Army  scientists  to  contribute  many  papers  for  both  the 
technical  and  clinical  sessions.  In  addition,  some  person  should  be  asked 
to  give  a history  of  these  conferences  and  their  importance  to  the  statistics 
used  by  the  Army.  This  individual  should  point  out  the  role  played  in  these 
conferences  by  Professor  Samuel  S.  Wilks,  and  also  discuss  his  many 
contributions  to  the  Army  and  to  the  other  armed  services.  The  Chairman  of 
the  Subcommittee,  Dr.  Walter  Foster,  reported  that  the  coming  conference 
would  probably  be  held  at  Fort  Belvoir  and  he  hoped  for  confirmation  of  this 
in  the  near  future. 

In  a letter  under  date  of  20  February  1974,  Lieutenant  Colonel  Harold 
P.  Hoefekamp  issued  a formal  invitation  to  hold  the  conference  at  Fort 
Belvoir  on  23-25  October  1974.  We  quote  the  following  paragraph  from  his 
letter:  "The  Operational  Test  and  Evaluation  Agency  and  the  Engineer 

Center  considers  it  an  honor  to  host  the  Army's  20th  Design  Conference. 

Every  effort  will  be  made  to  insure  that  the  best  facilities  and  support 
are  made  available  for  this  historic  event.  Both  the  Operational  Test  and 
Evaluational  Agency  and  the  Engineer  Center  are  fully  aware  of  the  conference's 
significance,  not  only  to  the  Army's  scientific  community,  but  to  the  Army 
as  a whole."  The  sentiments  expressed  in  this  letter  certainly  guided  the 
hosts  in  their  handling  of  this  meeting  as  it  was  one  of  the  best  conferences 
in  this  series.  This  was  no  doubt  due  largely  to  the  expertise  with  which 
Fort  Belvoir  handled  the  arrangements  and  the  visitors.  Mr.  Walter  Hollis, 
Chairman  on  Local  Arrangements,  is  to  be  commended  on  a very  fine  job. 
Unfortunately , he  had  to  be  out  of  the  country  on  the  dates  of  the  conference. 
In  his  absence,  Captain  Stanley  Dahl  in  took  over  his  chores.  He  deserves 
special  recognition  for  the  manner  in  which  he  performed  his  assigned  duties. 

Each  year  the  Program  Committee  is  instructed  to  select  invited  speakers 
who  can  discuss  in  an  informative  and  stimulating  manner  statistical  areas 
of  current  interest.  At  least  one  of  the  speakers,  who  has  expertise  in 
areas  of  special  interest  to  the  host  installation,  is  asked  to  present 
new  developments  in  these  fields.  These  selection  criteria  were  certainly 
met  by  the  gentlemen  giving  the  talks  in  the  General  Sessions.  The  titles 
of  their  addresses  are  noted  below: 
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Samuel  $.  Wilks  and  the  Army  Experiment  Design  Conferences 
Dr.  Churchill  Eisenhart,  National  Bureau  of  Standards 

Multidimensional  Contingency  Tables 

Professor  Solomon  Kullback,  The  George  Washington  University 

Multivariate  Data  Analysis 

Professor  Herbert  Solomon,  Stanford  University 

Order  Statistics 

Professor  H.  A.  David,  Iowa  State  University 
Reliability 

Professor  Gerald  Lieberman,  Stanford  University 

Ranking  and  Selection  Procedures 

Professor  Robert  Bechhofer,  Cornell  University 

Maximum  Information  from  Experiments 

Dr.  Marion  R.  Bryson,  U.S.  Army  Combat  Development  Experiment  Command  and 
Dr.  William  Mallios,  McDonald  Service  Company 


The  tenth  Samuel  S.  Wilks  Memorial  Award  of  the  American  Statistical 
Association  was  presented  to  Mr.  Cuthbert  Daniel  for  his  many  outstanding 
contributions  to  the  applications  of  statistics.  The  presentation  of  the 
medal,  citation  and  honorarium  was  made  by  Professor  Jerome  Cornfield, 
President  of  the  American  Statistical  Association.  More  details  about  this 
award  appear  in  the  body  of  these  proceedings. 

Probably  the  most  valuable  phases  of  these  conferences  are  the  technical 
and  clinical  sessions.  In  the  technical  sessions  Army  scientists  announce 
their  successes  in  handling  a few  of  the  many  technical  problems  they  face, 
while  in  the  clinical  sessions  they  have  a chance  to  get  help  from  nationally 
known  scientists  on  ways  to  cope  with  some  of  their  unsolved  design 
problems.  This  year  there  were  thirty-four  (34)  technical  papers  and  eight 
(8)  clinical  papers  on  the  agenda.  We  are  pleased  to  be  able  to  print 
many  of  these  contributed  papers  in  this  technical  manual. 

Members  of  my  Program  Committee  (Marion  Bryson,  Gerard  Dobrindt, 

Walter  Foster,  Fred  Frishman  (Secretary),  Walter  Hollis,  Badrig  Kurkjian, 
Clifford  Maloney,  Herbert  Solomon,  Douglas  Tang  and  Robert  Thrall)  are 
due  my  thanks  for  outlining  the  main  events  of  this  meeting  and  for  selecting 
such  an  outstanding  list  of  invited  speakers,  I would  also  like  to  express 
my  appreciation  to  Francis  Dressel  for  serving  as  secretary  during  the 
final  phases  of  this  conference. 


FRANK  E.  GRUBBS 
Conference  Chairman 
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CHAIRMAN 

Paul  C.  Cox,  Quality  Assurance  Office,  U.S.  Army  White  Sands 
Missile  Range,  White  Sands  Missile  Range,  New  Mexico 
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0830-1000  TECHNICAL  SESSION  6 - (Cont'd) 

AIRCRAFT  ARMAMENT  FIRE  CONTROL  SENSITIVITY  ANALYSIS  USING  A 
PROGRAMMABLE  CALCULATOR 

Thomas  O.  Mcln+lre,  Methodology  and  Instrumentation  Division, 
U.S.  Army  Yuma  Proving  Ground,  Yuma  Proving  Ground,  Arizona 

APPLICATIONS  OF  A MORE  EFFICIENT  SEQUENTIAL  SENSITIVITY  TEST 
STRATEGY  AND  ESTIMATION  METHODOLOGY  TO  RELIABILITY  ASSESSMENTS 

Gertrude  We intraub.  Ammunition  Development  and  Engineering 
Directorate,  Concepts  and  Effectiveness  Division,  Picatinny 
Arsenal,  Dover,  New  Jersey 

STATISTICAL  ANALYSIS  AND  MODELING  OF  SENSITIVITY  AUGMENTATION  IN 
CUTANEOUS  COMMUNICATIONS 

Richard  J.  D'Accardi  and  H,  S.  Bennett,  U.S.  Army  Electronics 
Command,  Fort  Monmouth,  New  Jersey 

1000-1030  BREAK 

1030-1130  GENERAL  SESSION  II  - Auditorium  of  Humphrey’s  Hall 
CHAIRMAN 

Professor  Boyd  Harshbarger,  Department  of  Statistics,  Virginia 
Polytechnic  Institute  and  State  University,  Blacksburg,  Virginia 

MULTIVARIATE  DATA  ANALYSIS 

Professor  Herbert  Solomon,  Department  of  Statistics,  Stanford 
University,  Stanford,  California 

1130-1300  LUNCH  - Mackenzie  Hall 

1300-1415  TECHNICAL  SESSION  7 - Auditorium  of  Humphrey’s  Hall 
CHA I RMAN 

Richard  J.  D’Accardi,  U.S.  Army  Electronics  Command,  Fort 
Monmouth,  New  Jersey 

SKIP-LOT  PR0CE0URE  FORMULATION  USING  THE  SIMPLIFIED  MARKOV  CHAIN 
METHOD 

Richard  M.  Brugger,  RAM  Assessment  Division,  U.S.  Army  Armament 
Command,  Rock  Island,  Illinois 

SEMI  MARKOV  CHAINS  APPLIED  TO  CONTINUOUS  SAMPLING  FLANS 

David  L,  Arp,  Naval  Weapons  Center,  Chlna.Lake,  California 
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1 300“ 1 4 1 5 TECHNICAL  SESSION  8 
CHA I RMAN 

Gertrude  Weintraub,  Ammunition  Development  and  Engineering 
Directorate,  Concepts  and  Effectiveness  Division,  Picatinny 
Arsenal,  Dover,  New  Jersey 

PREDICTING  RELIABILITY  GROWTH 

Larry  H.  Crow,  U.S.  Army  Materiel  Systems  Analysis  Agency, 
Aberdeen  Proving  Ground,  Maryland 

MINIMUM  VARIANCE  SOLUTION  OF  A POLYNOMIAL  FUNCTION  OF  TWO  NOISY 
RANDOM  VARIABLES 

Oren  N,  Dalton,  Mathematics  Services  Branch,  U.S.  Army  White 
Sands  Missile  Range,  White  Sands  Missile  Range,  New  Mexico 

1300-1415  TECHNICAL  SESSION  9 
CHA I RMAN 

OPT  Lolt  D.  Prolgler,  Analysis  Branch,  Technical  Support  Division 
U.S.  Army  Operational  Test  and  Evaluation  Agency,  Ft.  Bel  voir, 
Virginia 

THE  PROBABILITY  OF  MOTOR  CASE  RUPTURE 

Ronald  S.  Downs  and  Paul  C.  Cox,  Quality  Assurance  Office,  U.S. 
Army  White  Sands  Missile  Range,  White  Sands  Missile  Range, 

New  Mexico 

ON  THE  NONEXISTENCE  OF  SOME  INCOMPLETE  BLOCK  DESIGNS 

Alan  W.  Benton,  U.S.  Army  Materiel  Systems  Analysis  Agency, 
Aberdeen  Proving  Ground,  Maryland 


1415-1445  BREAK 

1445-1700  GENERAL  SESSION  III  - Auditorium  of  Humphrey’s  Hall 
CHA I RMAN 

Professor  Herbert  Solomon,  Department  of  Statistics,  Stanford 
University,  Stanford,  California 

ORDER  STATISTICS 

Professor  H.  A.  David,  Department  of  Statistics,  Iowa  State 
University,  Ames,  Iowa 

RELIABILITY 

Professor  Gerald  Lleberman,  Department  of  Operations  Research, 
Stanford  University,  Stanford,  California 
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0830-0915  TECHNICAL  SESSION  10  - Auditorium  of  Humphrey’s  Hall 
CHAIRMAN 

Boyd  Harshbarger,  Department  of  Statistics,  Virginia  Polytechnics 
Institute  and  State  University,  Blacksburg,  Virginia 

SIMPLE  STATISTICAL  ALTERNATIVES  TO  THE  METHOD  OF  LEAST  SQUARES 
FOR  THE  DETERMINATION  OF  X-INTERCEPT,  AND  SLOPE 

Joseph  F.  Hannigan,  Research  Institute,  U.S.  Army  Engineering 
Test  Center,  Fort  Be  I voir,  Virginia 

A STATISTICAL  APPROACH  TO  THE  LOADING  AND  FAILURE  OF  STRUCTURES 

Ronald  Merritt,  Construction  Engineering  Research  Laboratory, 
Champaign,  Illinois 

0830-0915  TECHNICAL  SESSION  II 
CHA I RMAN 

Lang  P.  Withers,  Analysis  Branch,  Technical  Support  Division, 

U.S.  Army  Operational  Test  and  Evaluation  Agency,  Fort 
Be  I vo i r , Virginia 

STRAIN  GAGE  INSTRUMENTATION  FOR  AMMUNITION  TESTING 

Paul  D.  Flynn,  Pitman-Dunn  Laboratory,  Frankford  Arsenal, 
Philadelphia,  Pennsylvania 

DATA  ANALYSIS  OF  AUTOMATIC  TRACKER  TESTING 

Nicholas  P.  Marasco,  Robert  T.  Volz  and  Thomas  G.  Kelley, 
Photoelectric  Branch,  Fire  Control  Development  and  Engineering 
Directorate,  Frankford  Arsenal,  Philadelphia,  Pennsylvania 

0830-0915  TECHNICAL  SESSION  12 
CHAIRMAN 

Roy  C.  Schmidt,  Test  Design  Division,  U.S.  Army  Operational  Test 
and  Evaluation  Agency,  Fort  Bel  voir,  Virginia 

STATISTICAL  INVESTIGATION  INTO  PULSE  CHARGING  OF  N1CKELCADMIUM 
BATTERIES 

Walter  Kastan  and  Erwin  Biser,  U.S.  Army  Electronics  Command, 

Fort  Monmouth,  New  Jersey 

OPTICAL  CHARACTERIZATION  OF  SURFACE  ROUGHNESS 

E.  L.  Church  and  J.  M,  Zavada,  Frankford  Arsenal,  Philadelphia, 
Pennsylvania 
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0915-1045  GENERAL  SESSION  IV  - Auditorium  of  Humphrey’s  Hall 
CHAIRMAN 

Dr.  Frank  E.  Grubbs,  Chairman  of  the  Conference,  U.S.  Army 
Ballistic  Research  Laboratories,  Aberdeen  Proving  Ground, 

Mary  I and 

OPEN  MEETING  OF  THE  AMSC  SUB-COMMITTEE  ON  PROBABILITY  AND 
STATISTICS 

Dr.  Walter  D.  Foster,  Computer  Services  Division,  Armed  Forces 
Institute  of  Pathology,  Washington,  D.C. 

RANKING  AND  SELECTION  PROCEDURES 

Professor  Robert  Bechhofer,  Department  of  Operations  Research, 
Cornell  University,  Ithaca,  New  York 

1 045- 1115  BREAK 

I I 1 5- 1 2 1 5 GENERAL  SESSION  IV  (continued) 

MAXIMUM  INFORMATION  FROM  EXPERIMENTS 

Dr.  Marion  R.  Bryson,  U.S.  Army  Combat  Developments  Experiments 
Command,  Fort  Ord,  California 

Dr.  William  Mallios,  Braddock,  Dunn,  McDonald  Service  Company, 
Fort  Ord,  California 
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SAMUEL  S.  WILKS  AND  THE  ARMY  EXPERIMENT  DESIGN  CONFERENCE  SERIES 


Churchill  Eisenhart 
Senior  Research  Fellow 
Institute  for  Basic  Standards 
National  Bureau  of  Standards 
Washington,  D.C.  20234 


ABSTRACT . A biography  of  Professor  Samuel  Stanley  Wilks  (1906-1964) 
of  Princeton  University,  with  particular  attention  to  his  early  life, 
notes  on  the  persons  who  shaped  his  professional  development,  review 
of  his  many  facetted  professional  career  and  his  role  In  initiating 
and  launching  the  U.S.  Army’s  annual  series  of  Conferences  on  the 
Design  of  Experiments  in  Army  Research,  Development  and  Testing. 

1.  BIRTH,  FAMILY,  AND  EARLY  YEARS.  Sam  Wilks  was  bom  on  the 
17th  of  June  1906  in  Little  Elm,  Denton  County,  Texas,  the  first  of 
the  three  children  of  Chance  C.  and  Bertha  May  Gammon  Wilks.  His  father 
trained  for  a career  in  banking,  but  after  a few  years  chose  instead  to 
make  his  livelihood  by  operating  a 250-acre  farm  near  Little  Elm.  His 
mother  had  a talent  for  music  and  art;  and  a lively  curiosity,  which 
she  transmitted  to  her  three  sons.  The  predilection  of  their  father. 
Chance  C.  Wilks,  for  alliteration  is  manifest  in  the  given  names  of  all 
three:  ^amuel  Stanley,  Syrrel  Singleton,  and  William  Weldon  (Wilks). 

Syrrel,  less  than  two  years  younger  than  Sam,  was  his  boyhood 
companion;  studied  biology  (B.S.,  1927)  and  physiology  (Ph.D.,  1936); 
became  Associate  Professor  of  Physiology  at  the  Air  Force  School  of 
Aviation  Medicine;  and  passed  away  early  this  year  (1974).  In  consequence 
of  Sam's  and  Syrrel' s initials  being  the  same,  their  publications  are 
sometimes  lumped  together  under  "S.  S.  Wilks"  in  bibliographic  tools, 
e.g.,  in  the  successive  volumes  of  the  Science  Citation  Index. 

Sam's  "baby  brother,"  William,  was  eight  years  younger.  He  also 
took  a B.S,  degree;  became  a research  advisor  to  Bell  Aircraft  Company 
in  Fort  Worth,  Texas;  and  is  still  living.  The  choice  of  "Weldon" 
for  his  middle  name  is  merely  a happenstance  of  his  father's  effort 
to  achieve  a triple  alliteration,  and  has  no  genealogical  significance: 
there  is  no  known  connection  between  the  Chance  Wilks  family  and  that 
of  the  pioneer  biometrician,  W.  F.  R.  Weldon  (1860-1906),  who  died  in 
London  in  April  of  the  year  in  which  our  blographee  was  bom;  or  with 
any  other  Weldons. 

Sam  began  his  early  education  in  a typical  one-room  rural  school 
house  where,  in  the  seventh  grade,  he  had  as  his  teacher  William  Marvin 
Whyburn,  who  became  a distinguished  mathematician,  the  president  of 
Texas  Technological  College  (1944-1948) , and  the  chairman  or  head  of 


two  university  departments  of  mathematics  (UCLA,  1937-1944;  University 
of  North  Carolina,  1948-1956,  1960-1965) — the  first  of  an  extraordinary 
number  of  prominent  people  who  had  a part  in  Sam’s  education.  He 
attended  high  school  in  Denton,  the  county  seat  and  the  site  of  North 
Texas  State  Teachers  College  (now  North  Texas  State  University) , and 
of  a College  of  Industrial  Arts  for  women  (now  Texas  Woman's  University). 
During  the  week  he  roomed  in  Denton,  and  went  home  on  weekends,  walking 
the  15  miles  to  his  father’s  ranch  when  necessary.  During  his  final 
year  of  high  school,  it  was  noticed  that  he  was  absent  repeatedly  from 
study  hall.  Inquiry  revealed  that  he  was  skipping  study  hall  in  order 
to  take  a mathematics  course  at  North  Texas  State  Teachers  College. 

Following  graduation  from  high  school,  Sam  continued  his  studies 
at  North  Texas  State  Teachers  College,  where  he  followed  an  industrial 
arts  program,  with  particular  attention  to  mathematics.  He  received 
an  A.B.  in  architecture  in  June  1926,  a few  days  before  his  20th  birth- 
day. A large  drinking  fountain,  designed  by  Sam  and  a friend,  on  the 
campus  of  the  College  attests  to  his  talent  and  serves  as  a reminder 
of  his  one-time  interest  in  architecture.  But  believing  his  eyesight 
inadequate  for  the  life  of  an  architect,  he  turned  to  a career  in 
mathematics, 

2,  TEACHING  AND  GRADUATE  STUDY,  During  the  school  year  1926-1927, 
Wilks  taught  mathematics  and  manual  training  in  a public  school  in  Austin, 
Texas,  and  began  graduate  study  of  mathematics  at  the  University  of  Texas 
there.  He  continued  his  studies  at  the  University  of  Texas  as  a part-time 
instructor  in  mathematics  1927-1928;  and  received  an  M.A.  in  mathematics 
in  1928.  His  first  course  in  advanced  mathematics  at  the  University  of 
Texas  was  set  theory,  taught  by  R.  L,  Moore  (1882-1974),  renowned 
among  mathematicians  for  his  research  in  topology,  his  unusual  methods 
of  teaching,  and  the  vigor  and  resoluteness  of  his  opinions.  Wilks 
was  fascinated  by  the  unfolding  of  this  beautiful  theory  from  a few 
simple  definitions  and  axioms,  but  Moore's  espousal  of  pure  mathematics 
as  a discipline  wholly  divorced  from  application,  and  Moore's  scorn  of 
applied  mathematics  as  work  on  a level  with  washing  dishes,  were 
incomprehensible  and  unacceptable  to  him.  Had  Moore's  attitude  been^, 
otherwise,  Sam  might  have  become  a topologist.  But,  as  Alex  M,  Mood— / 
has  said  in  his  note  on  Sam's  philosophy  about  his  work,  "Sam’s  character 
demanded  that  his  work  be  immediately  and  obviously  useful  [andj  Moore 
was  the  last  man  to  persuade  him  that  point  set  theory  was  useful." 

(MOOD  1965,  p.  953)  Much  more  in  keeping  with  his  "character"  were 
probability  and  statistics,  to  which  he  was  introduced  by  Edward  L. 

Dodd  (1875-1943),  an  inspiring  teacher  and  distinguished  scholar, 
noted  for  his  researches  . on  mathematical  and  statistical  properties  of 
various  types  of  means,— 

An  aside  on  Sam's  views  with  respect  to  pure  mathematics  and  pure 
mathematicians  seems  appropriate  at  this  juncture,  before  taking  up  the 
next  step  in  his  education.  To  this  end  I can  do  no  better  than  to 
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quote  further  from  Mood’s  note: 

’’Wilks, . .saw  little  sense  in  pure  mathematics  unless  it 
had  some  ultimate  application.  He  generally  believed 
that  most  pure  mathematics  would  eventually  justify  itself 
in  this  way  and  was  delighted  when  that  did  happen  in  his 
own  work  or  that  of  others ... .The  set  theoretical  founda- 
tion of  probability  theory  developed  by  Kolmogorov  gave 
Sam  no  end  of  pleasure  partly  because  of  that  early  course, 
perhaps,  but  more  likely  because  it  was  a good  piece  of 
evidence  that  pure  mathematicians  were  not,  after  all* 
wasting  their  time. 

"While  Sam  was  generally  optimistic  about  the  eventual 
utility  of  pure  mathematics  he  became  less  and  less 
patient  over  the  years  with  pure  mathematicians  themselves — 
especially  those  in  the  United  States.  For  one  thing  he 
believed  that  their  general  refusal  to  apply  their  intellects 
even  briefly  to  important  practical  problems  was  less  than 
patriotic,  to  say  the  least.  He  rarely  missed  an  opportunity 
to  point  out  that  almost  all  top-level  Soviet  mathematicians 
had  at  one  time  or  another  turned  to  an  important  field  of 
application  thus  placing  themselves,  in  his  eyes,  quite  above 
many  of  America’s  leading  mathematicians. 

"The  thing  that  particularly  annoyed  Sam  about  pure 
mathematicians  was  their  snobbishness  about  pure  mathematics 
and,  worse,  their  success  in  generating  the  same  sort  of 
snobbishness  in  every  mathematically  talented  student  that 
came  along.  Sam  was  a very  even  tempered  man  but  this  was 
a subject  that  could  summon  loud  indignation  from  him.  He 
believed  that  for  reasonably  even  balance  in  the  development  of 
mathematics  a substantial  proportion  of  the  most  talented 
students  should  go  into  mathematical  statistics,  mathematical 
physics,  applied  mathematics,  econometrics,  etc.  As  it  was, 
he  believed  that  pure  mathematics  preempted  over  nine  out 
of  ten  of  the  most  talented  students  thus  completely  deforming 
mathematical  progress  in  the  United  States.  In  his  later 
years  he  maintained  that  it  was  impossible  for  him  to  persuade 
enough  sufficiently  promising  college  graduates  to  undertake 
work  in  statistics  at  Princeton  and  therefore  he  had  to  go  to 
Britain  and  Canada  to  find  good  students  whose  attitudes  had 
not  been  corrupted  by  pure  mathematicians  in  the  United 
States."  (MOOD  1965,  953-954) 

When  Sam  completed  the  requirements  for  his  M.A.  in  mathematics 
at  the  University  of  Texas  in  1928,  Professor  Dodd  encouraged  him  to 
pursue  further  study  of  mathematical  statistics  at  the  University  of 
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Iowa—  under  Henry  L,  Rietz  (1875-1943)—  , the  leader  of  his  generation 

in  American  mathematical  statistics.-  Wilks  stayed  on  at  the  Univer- 
sity of  Texas  as  an  instructor  in  mathematics  during  the  summer  of  1928, 
and  the  academic  year  1928-29;  applied  for  a fellowship  at  the  University 
of  Iowa;  and  to  pick  up  some  ready  cash,  served  as  a monitor  for  State 
bar  exams  given  at  the  University. 

In  due  course,  Sam  was  offered,  and  accepted,  a fellowship  at  the 
University  of  Iowa,  in  Iowa  City.  He  arrived  in  Iowa  City  in  the 
summer  of  1929  to  begin  a two-year  program  of  graduate  study  and  research 
leading  to  a Ph.D,  degree  in  mathematics,  with  a minor  in  education. 
During  the  second  summer  (1930),  he  was  joined  by  two  others  whose 
names  were  later  to  become  well-known  in  probability  and  mathematical 
statistics  circles:  Allen  T.  Craig  and  John  H.  Curtiss. 

Curtiss  had  just  received  his  A.B.  in  mathematics  at  Northwestern 
University,  and  had  come  to  the  University  of  Iowa  to  study  actuarial 
mathematics  preparatory  to  choosing  actuarial  work  as  a career.  He 
was  assigned  to  one  of  two  desks  arranged  back-to-back  in  the  Mathe- 
matics Department  Library,  the  other  occupied  by  gam.  He  has  a close- 
up  picture  of  Sam  taken  from  this  vantage  point.— 

Allen  T.  Craig,  in  contrast,  had  returned  to  the  University  of 
Iowa  in  the  summer  of  1930  for  the  express  purpose  of  completing  his 
doctoral  thesis  "On  the  Distribution  of  Certain  Statistics  Derived 
from  Small  Random  Samples".  I say  "had  returned  to  the  University 
of  Iowa"  because  Craig  had  been  there  during  the  academic  year  1928-29, 
but  had  left  Iowa  City  in  the  summer  of  1929  to  accept  a position  as 
an  Instructor  In  mathematics  at  his  alma  mater,  the  University  of 
Florida,  in  Gainesville,  for  the  academic  year  1929-30.  Drawn  together 
by  common  Interests,  Allen  Craig  and  Sam  Wilks  immediately  became 
close  and  lifelong  friends.  Craig,  in  his  thesis  (CRAIG  1932),  gave 
a number  of  general  results  on  the  distributions  of  such  statistics 
as  the  arithmetic  mean,  harmonic  mean,  geometric  mean,  median,  quartile, 
decile  and  range  of  samples  of  small  ii  items  selected  at  random  from 
a rather  arbitrary  (continuous)  universe,  together  with  a large  number 
of  explicit  results  for  sampled  universes  of  special  types.  Sam 
often  said  that  his  own  work  on  the  theory  of  nonparametric  or  distri- 
bution-free methods — an  area  in  which  Sam  made  a number  of  truly 
outstanding  contributions^  — had  its  origins  in  the  general  formulas 
given  by  Craig  for  the  distributions  of  the  ''median,  quartile,  decile, 
and  range". 

Sam's  doctoral  dissertation  was,  likewise,  a contribution  to  "the 
theory  of  small  samples".  Entitled  "On  the  distributions  of  statistics 
in  samples  from  a normal  population  of  two  variables  with  matched 
sampling  of  one  variable"  (WILKS  1932a),  it  provided  the  small-sample 
distribution  theory  required  to  answer  a number  of  questions  drawn  to 
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Sam’s  attention  by  Professor  E.  F.  Lindquist,  Professor  of  Education 
at  the  University  of  Iowa  and  Director  of  the  Iowa  Testing  Programs, 
who  had  used  the  technique  of  "matched"  groups  in  experimental  work 
in  educational  psychology,  and  whose  lectures  Sam  had  attended. 

Sam's  thesis  was  preceded  by  a short  note  by  Sam  on  "The  standard 
error  of  the  means  of  ’matched'  samples"  (WILKS  1931),  published  in 
the  March  1931  issue  of  the  Journal  of  Educational  Psychology,  where 
it  was  accompanied  by  an  article  by  Lindquist  (LINDQUIST  1931), 
describing  the  use  and  importance  of  "matched"  groups  as  a statistical 
technique  in  experimental  psychology  and  educational  testing.  Sam's 
predoctoral  note  and  his  doctoral  dissertation  were  the  first  of  a 
series  of  papers  on  multivariate  analysis  suggested  by  real-life 
problems  in  experimental  psychology  and  educational  testing,  and  mark 
the  beginning  of  Sam’s  life-long  association  with  the  latter  field. 

Sam  and  Allen  Craig  both  received  their  Ph.D.’s  from  the  University 
of  Iowa  in  June  1931 — Sam  in  Mathematics,  with  a minor  in  Education; 

Allen,  in  Mathematics  alone.  "Father  Rietz"  was  mighty  proud  of  his 
"twins".  Theirs  were  the  first  doctoral  dissertations  written  at  the 
University  of  Iowa  on  aspects  of  "the  theory  of  small  samples",  the 
new  area  of  mathematical  research.  Initiated  in  1908  by  "Student" 

(William  Sealy  Gosset,  1876-1937)  and  developed  to  full  flower  by 
R.  A.  Fisher  (1890-1962)  between  1915  and  1928,  to  which  an  increasing 
number  of  American  mathematicians  were  devoting  attention  at  that 
time — notably  C.  C.  Craig  (at  the  University  of  Michigan  in  Ann  Arbor), 
Harold  Hotelling  (at  Stanford  University,  in  California),  Paul  R.  Rider 
(at  Washington  University,  St,  Louis),  and  Rietz  (at  the  University  of 
Iowa,  in  Iowa  City) & Rietz  was  doubly  proud  of  their  accomplishments; 
not  only  had  each  made  a first-rate  contribution  to  "the  theory  of 
small  samples",  but  also  the  mathematics  in  their  dissertations  was 
intelligible  to  American  mathematicians — which  was  a great  deal  more 
than  one  could  say  about  the  papers  of  R.  A.  Fisher.^  He  therefore 
held  out  two  "prizes"  to  his  deserving  "twins":  (1)  an  appointment  as 
an  Associate  (a  rank  between  Instructor  and  Assistant  Professor) 
in  his  department,  and  (2)  his  endorsement  for  a National  Research 
Council  Fellowship.  Allen  chose  the  appointment  in  the  Department  of 
Mathematics — stayed  on  to  become  a full  Professor  in  1945,  and  retired 
in  1970;  Sam,  the  NRC  Fellowship,  and  made  plans  to  continue  research 
in  multivariate  statistical  analysis  under  Harold  Hotelling  (1895-1973), 
a pioneer  in  this  field,  and  the  individual  in  the  United  States  most 
versed  in  the  mathematics  of  the  Student-Fisher  theory  of  small  samples. — 

After  receiving  his  Ph.D.,  Wilks  stayed  on  to  attend  the  lectures 
given,  and  seminar  conducted,  during  the  first  half  of  the  summer 
session,  8 June  - 16  July,  by  the  British  mathematical  statistician. 
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Egon  S,  Pearson;  and  gave  a talk  in  the  seminar  series,  Pearson’s 
two  papers  with  Jerzy  Neyman  on  "The  use  and  interpretation  of  certain 
test  criteria  for  the  purposes  of  statistical  inference"  (Part  I, 
Biometrika,  Vol.  20A  (1928),  pp.  175-240;  Part  II,  ibid,  pp.  263-294) 
had  been  well  received  by  mathematicians  interested  in  statistical 
theory.  As  you  will  recall,  it  was  in  these  papers  that  they  introduced 
and  explored  their  likelihood-ratio  technique  for  more  or  less  ^ 

automatically  discovering  "good"  tests  of  various  statistical  hypotheses. — J 

Wilks  also  met  R,  A.  Fisher,  who  came  over  to  Iowa  City  from  Ames 
for  a day  during  this  period.  By  an  extraordinary  coincidence,  R.  A, 

Fisher  was  "in  residence"  that  summer  at  Iowa  State  College,  at  Ames, 

90  miles  distant,  giving  a "competing"  series  of  lectures  on  the 
material  in  his  two  books , Statistical  Methods  for  Research  Workers 
(3rd  edition,  1930)  and  The  Genetical  Theory  of  Natural  Selection  (1930), 
during  the  first  half  of  their  summer  session,  16  June-24  July.  The 
overlap  of  the  two  programs,  and  the  distance  between  the  two  Institutions, 
made  it  physically  impossible  for  faculty  and  students  to  take  in  both 
programs  in  their  entirety. 


3.  MARRIAGE  AND  POSTDOCTORAL  STUDY.  Sam  returned  to  Texas  in 
midsummer  1931,  and  on  September  1 married  Gena  Orr  of  Denton.  The 
Wilks  and  Orr  families  had  been  friends  for  many  years.  Indeed,  about 
one  year  before  Chance  Wilks  finally  won  the  hand  of  Bertha,  she  was 
being  courted  by  Will  Orr,  while  Chance  was  away  from  Little  Elm, 
trying  his  hand  at  the  banking  business.  But  Chance  returned  in  time 
to  prevent  my  story  from  ending  before  it  began — and  in  due  course  Gena 
was  fathered  by  Will;  and  Sam,  by  Chance. 

Sam  and  Gena  had  known  each  other  from  childhood.  They  attended 
the  same  high  school  in  Denton;  she  was  a student  at  the  College  of 
Industrial  Arts,  in  Denton,  at  the  same  time  that  Sam  was  attending 
the  North  Texas  State  Teachers  College  there;  and  they  both  received 
their  A.B.  degrees  In  1926;  but  they  did  not  start  "dating"  until 
that  summer.  What  brought  them  together  was  the  wedding  of  Sam's 
cousin,  James  Hodge,  and  Jessie  Hill,  at  which  Sam  was  Best  Man,  and 
Gena  a bridesmaid.  Gena  then  taught  school  locally  for  a couple  of 
years,  while  Sam  was  continuing  his  study  of  mathematics  at  the 
University  of  Texas,  In  Austin;  and  continued  to  date  Sam  from  time 
to  time  when  he  was  home  on  vacation.  In  due  course  she  got  herself 
over  to  the  University  of  Texas,  where  she  did  graduate  work  in 
English,  and  received  her  Master's  Degree  in  1929. 
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As  part  of  their  honeymoon,  Sam  and  Gena  set  off  for  New  York 
City  by  boat,  from  Galveston,  Texas.  The  trip  took  five  days.  They 
settled  in  an  apartment  on  the  6th  floor  of  the  Columbia  University- 
owned  apartment  building  at  401  West  118th  Street.  During  World  War  II 
the  main  offices  of  the  Statistical  Research  Group — Columbia  (SRG-C) » 
of  which  Harold  Hotelling  was  the  Principal  Investigator,  were  located 
in  this  building,  in  what  had  been  Sam  and  Gena’s  apartment;  and  W. 

Allen  Wallis,  the  Group’s  Director  of  Research,  occupied  what  had  been 
their  bedroom. 

Among  those  attending  Hotelling's  lectures  on  "Statistical 
Inference"  that  first  year  at  Columbia  in  addition  to  Sam  were  Acheson 
J.  Duncan,  from  whom  X was  later  to  receive  my  first  course  in  this 
subject,  and  W.  J.  Youden  (1900-1971),  who  was  later  to  join  me  at 
the  National  Bureau  of  Standards  (1948-1965)  as  practitioner,  expositor 
and  innovator  of  statistical  methods  par  excellence.  "Atch"  Duncan 
was  then  an  Instructor  in  Economics  at  Princeton  University,  and  at 
my  father’s  insistence  had  been  sent  at  University  expense  to  study 
modern  statistical  inference  under  Hotelling.  X shall  say  more  about 
this  in  a few  moments.  "Jack"  Youden  had  received  his  Ph.D.  in 
Chemistry  from  Columbia  in  1924,  was  a Physical  Chemist  at 
the  Boyce  Thompson  Institute  for  Plant  Research  in  Yonkers,  New  York, 
and  was  commuting  to  New  York  to  hear  Hotelling's  lectures  on  his  own-„/ 
volition  to  gain  a better  grasp  of  Student-Fisher  theory  and  methods. — 

In  addition  to  auditing  Hotelling’s  lectures,  Sam  joined  Jack  W. 
Dunlap  and  Warren  G.  Findley,  then  Ph.D.  candidates  at  Columbia  In 
Psychology  and  Educational  Psychology,  respectively,  in  attending 
the  lectures,  at  Teachers  College,  of  the  English  psychologist, 

Charles  E.  Spearman  (1863-1945),  revered  by  psychologists  as  the  father 
of  Factor  Analysis  (1904)  and  for  development  of  a rational  basis  for 
determining  general  intelligence  and  for  validating  intelligence 
testing. ^3/  I mention  Jack  Dunlap  and  Warren  Findley  explicitly 
because  Sam's  and  their  paths  were  to  meet  and  join  for  a while  at 
various  times  in  later  years,  for  example,  when  Dunlap  was  Director 
of  Research  of  the  National  Research  Council’s  Committee  on  Pilot 
Selection  and  Training  (1941-42) , and  when  Findley  was  Director  of 
Test  Development  (1948-53),  and  later  in  charge  of  the  Evaluation  and 
Advisory  Services  (1953-56)  of  the  Educational  Testing  Service  in 
Princeton. 

It  was  a year  of  exceptional  productivity  for  Wilks:  he  wrote 

or  completed  four  distinct  papers  in  the  area  of  multivariate  analysis 
all  of  which  saw  almost  immediate  publication.  In  one  (WILKS  1932b) 
he  found  the  maximum  likelihood  estimates  of  the  parameters  of  a 
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bivariate  normal  distribution  when  some  of  the  individuals  in  a sample 
yield  observations  on  both  variables,  x and  y,  and  some  only  on  x,  or 
on  y,  alone;  in  a second  (WILKS  1932c),  he  showed  that  the  distribution 
of  the  multiple  correlation  coefficient  in  samples  from  a normal 
population  with  a non-zero  multiple  correlation  coefficient  could  be 
derived  directly  from  Wlshart's  generalized  product  moment  distribution 
(1928)  without  making  use  of  the  geometrical  notions  and  an  invariance 
property  utilized  by  R.  A.  Fisher  in  his  derivation  (1928) ; in  the 
third,  his  great  paper  on  "Certain  Generalizations  in  the  Analysis 
of  Variance"  (WILKS  1932e),  he  defined  the  "generalized  variance"  of 
a sample  of  n individuals  from  a multivariate  population,  constructed 
multivariate  generalizations  of  the  correlation  ratio  and  coefficient 
of  multiple  correlation;  deduced  the  moments  of  the  sampling  distributions 
of  these  and  other  related  functions  in  random  samples  from  a normal 
multivariate  population  from  Wishart's  generalized  product  moment 
distribution  (1928) ; constructed  the  likelihood  ratio  criterion  for 
testing  the  null  hypothesis  that  k multivariate  samples  of  sizes  n^, 
n£,  — , nfc  are  random  samples  from  a common  multivariate  normal 
population,  now  called  "Wilks’s  A criterion",  and  derived  its  sampling 
distribution  under  the  null  hypothesis;  and  similarly  explored  various 
other  multivariate  likelihood  ratio  criteria;  and  in  the  fourth  (WILKS 
1932d) , an  outgrowth  of  attending  Spearman's  lectures,  he  obtained  an  ,,  , 
exact  expression  for  the  standard  error  of  an  observed  "tetrad  difference"— 
in  samples  of  size  n from  a normal  population  (in  the  special  case  in 
which  the  intercorrelations  of  the  four  variables  are  all  zero  in  the 
population) . 

I mention  these  details  just  to  show  to  what  a remarkable  extent 
Sam  was  not  only  applying,  but  also  extending  the  most  advanced  concepts 
and  tools  of  Fisher,  Hotelling,  Neyman,  E,  S.  Pearson  and  Wishart 
within  one  year  of  the  receipt  of  his  Ph.D. ! I often  heard  my  father, 

Luther  Pfahler  Eisenhart  (1876-1965),  remark  when  he  was  Chairman  of 
the  Department  of  Mathematics  (1928-1945)  and  Dean  of  the  Graduate 
School  (1933-1945)  of  Princeton  University,  that  what  determined  a 
man’s  stature  in  his  chosen  field  was  not  the  caliber  of  his  doctoral 
dissertation,  but  rather  the  caliber  of  the  papers  that  he  wrote  and 
published  after  receiving  his  Ph.D.  Sam  certainly  passed  that  test 
in  1932  with  a wide  margin  to  spare!  Furthermore,  the  high  regard  in 
which  Sam’s  papers  were  held  immediately  following  their  publication  is 
attested  by  the  fact,  already  mentioned,  that  Irwin  devoted  9 out  of  the 
14  pages  on  "Exact  sampling  distributions"  in  his  "Recent  Advances .. (1932 ) " 
(IRWIN  1934)  to  detailed  consideration  of  Sam's  thesis  and  the  first 
three  of  these  four  postdoctoral  papers.  And  E,  S.  Pearson  more 
recently  remarked  that  Sam's  "stature  as  a statistician  was  I think 
early  established  by  his  Biometrika  paper  of  1932  on  ’Certain  generalizations 
in  the  analysis  of  variance'  [which]  must  have  been  written  during  the 
winter  after  he  gained  his  Ph.D.  and  as  such  was  a remarkable 
performance."  (PEARSON  1964,  p.  597) 
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While  at  Columbia  University,  Sam  went  down  to  the  Bell  Telephone 
Laboratories  at  463  West  Street  to  visit  Walter  A,  Shewhart  (1891-1967), 
father  of  statistical  quality  control  of  manufacturing  processes, 
with  whose  work  he  had  become  acquainted  through  Rietz  and  Hotelling. — 

Sam  became  very  interested  in  Shewhart' s work,  and  shortly  thereafter 
Sam  and  Gena  paid  a brief  visit  to  Walter  and  Edna  Shewhart  at  their 
home  in  Mountain  Lakes,  New  Jersey.  Several  years  ago,  Mrs.  Shewhart 
told  me  that  she  remembered  well  how,  as  soon  as  Sam  and  Gena  had  left, 
Walter  had  turned  to  her  and  said,  "There  is  a young  man  who  is  going 
to  be  one  of  the  top  men  in  Statistics  in  this  country",  or  words  to 
that  effect.  This  was  the  beginning  of  the  friendship  and  collaboration 
of  these  two  men  that  continued  until  Sam's  death. 

In  the  Spring  of  1932,  Sam  obtained  a renewal  of  his  National 
Research  Fellowship,  as  an  International  Research  Fellowship.  He 
and  Gena  set  off  in  August  1932  for  London,  England,  where  Sam  was  to 
be  in  residence  in  Karl  Pearson's  Department  of  Applied  Statistics 
at  University  College  (of  the  University  of  London)  during  the 
"Michaelmas  Term"  (Sept .-Dec .) While  there,  Sam  and  Karl  Pearson's 
son,  Egon  S.  Pearson,  wrote  a joint  paper  (PEARSON  and  WILKS  1933)  in 
which  the  likelihood  ratio  techniques  of  Sam's  generalized  analysis-of- 
variance  paper  are  developed  in  greater  detail  for  samples  from  a 
bivariate  normal  distribution,  generalizing  to  this  bivariate  case  the 
three  tests  developed  by  Neyman  and  Pearson  (1931)  .for  the  univariate 
case.  To  illustrate  the  numerical  application  of  the  procedures  they 
had  developed,  they  included  two  worked  examples,  one  based  on  data 
on  the  tensile  strength  and  Rockwell  hardness  of  aluminum  dicastings, 
taken  from  Walter  A.  Shewhart1 s Economic  Control  of  Quality  of  Manufactured 
Product  (1931) 

While  in  London,  Sam  met  a great  many  of  the  leading  British 
statisticians,  and  their  disciples,  either  at  University  College  or  at 
the  delightful  teas  that  preceded  the  monthly  meetings  of  the  Royal 
Statistical  Society,  To  add  to  the  excitement — and  to  the  strain  of 
a married  couple's  attempting  to  live  in  London  on  the  small  stipend 
of  an  International  Research  Fellow — Sam  and  Gena's  son  Stanley  Neal 
Wilks  was  born  in  London,  in  October  1932.^/  Early  in  January  1933, 
the  family  of  three  moved  to  Cambridge  so  that  Sam  could  work  with 
John  Wishart  (1898-1956).  whose  work  in  multivariate  analysis  was  close 
to  Sam's  main  in  teres  t.-“/ 

When  Sam  arrived  at  Cambridge,  he  found  that  Wishart  and  Bartlett 
had  just  completed  an  "independent"  derivation  of  Wishart 's  generalized 
product-moment  distribution  "by  purely  algebraic  methods",  that  is, 
by  means  of  moment-generating  functions  in  combination  with  the  matrix 
algebra  of  quadratic  forms  (WISHART  and  BARTLETT  1933).  Wilks  found 
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himself  right  at  home  in  their  company,  and  promptly  wrote  another 
major  paper  (WILKS  1934)  in  which  he  gave  a method  of  deriving  directly 
from  the  multivariate  normal  distribution  (l.e, , without  using  the 
Wishart  distribution)  the  moments  of  the  sampling  distributions  of 
functions  of  determinants  of  the  types  considered  in  his  two  Biometrika 
papers.  Also,  at  the  suggestion  of  G.  Udny  Yule  (1864-1951),  he  wrote 
a paper,  "On  the  Independence  of  Sums  of  Squares  in  the  Analysis  of 
Variance",  in  which  by  means  of  characteristic  functions  in  combination 
with  elementary  matrix  algebra,  he  demonstrated  the  independence  of 
various  row,  column,  etc.,  "sums  of  squares"  involved  an  analysis-of- 
variance  analysis  of  randomized  blocks,  Latin  square,  and  certain  other 
experimental  arrangements,  discussed  previously  by  R.  A.  Fisher. 
Communicated  to  the  Royal  Society — not  the  Royal  Statistical  Society— 
by  Yule,  for  publication  in  its  Proceedings , the  paper  suffered  rough 
treatment i it  was  apparently  sent  to  Fisher  to  referee,  who  seems 
to  have  felt  that  by  Its  very  theme  it  implied  that  he  had  not  already 
given  adequate  and  intelligible  proofs;  then  the  manuscript  was  lost, 
and  Sam  had  to  provide  a second  copy;  and  then  it  was  rejected.  The 
publication  shortly  thereafter,  in  a publication  of  the  Royal  Statistical 
Society,  of  a similar,  but  somewhat  more  elementary,  paper  on  the 
same  subject,  by  one  of  Fisher’s  proteges,  was  a sore  point  with  Sam 
for  many  years.  (I  have  discussed  this  matter  with  the  author  of  the 
"offending"  paper.  He  assures  me  that  he  never  saw  Sam’s  manuscript; 
and,  until  our  conversation,  never  knew  of  its  existence.) 


In  Hay  1933  my  father  offered  Sam  an  appointment  as  an  Instructor 
in  Mathematics  in  Princeton  University.  I first  met  Sam  when  he  turned 
up  in  Princeton  in  time  for  the  fall  semester  1933,  imported  for  the 
express  purpose  of  teaching  me — at  least,  that  was  what  I thought  at 
the  time.  My  budding  interest  in  probability  and  statistics  may  have 
helped  a tiny  weeny  bit,  but  the  true  explanation  was  quite  otherwise, 
and  has  an  interesting  background, 

4.  WILKS’S  PRINCETON  APPOINTMENT,  AND  STATISTICS  AT  PRINCETON 
BEFORE  WILKS , The  key  figure  in  Wilks's  appointment  was  my  father, 
Luther  Pfahler  Eisenhart  (1876-1965),  who,  in  the  spring  of  1933,  was 
not  only  willing,  but,  as  Chairman  of  the  Department  of  Mathematics 
(1928-1945),  Dean  of  the  Faculty  (1925-1933),  and  Chairman  of  the 
University  Committee  on  Scientific  Research  (1930-1945),  was  also 
able  to  effect  Wilks's  appointment  to  an  Ins  true torship  In  Mathematics 
on  a more  or  less  emergency  basis  over  the  opposition  of  almost  every 
member  of  his  Department 

An  event  that  was  to  be  instrumental  in  bringing  both  mathematical 
economics  and  modern  statistical  theory  and  methodology  to  the  Princeton 
campus  was  the  arrival  of  Charles  F,  Roos  (1901-1958)  as  a National 
Research  Fellow  in  Mathematics  for  the  academic  year  1927-28.  Roos  had 
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received  his  Ph.D.  in  theoretical  economics  at  the  Rice  Institute  in 
1926  under  Professor  G.  C,  Evans  (1887-1973),  who  at  that  time  was 
developing  a new  mathematical  theory  of  economic  phenomena  termed 
"economic  dynamics",  and  had  spent  1926-27  at  the  University  of  Chicago 
working  with  Professor  Henry  Schultz  (1893-1938)  who  at  that  time  was 
deeply  engaged  in  his  epochal  research  on  statistical  laws  of  demand 
and  supply  as  one  facet  of  his  life's  work  on  the  theory  and  measurement 
of  demand.  Roos  came  to  Princeton  primarily  to  broaden  and  sharpen 
his  knowledge  of  mathematics  as  a basis  for  making  further  contributions 
to  Professor  Evans'  new  "economic  dynamics".  While  there  he  succeeded 
in  convincing  some  members  of  the  Department  of  Economics  and  Social 
Institutions  that  the  Department  could  not  afford  to  continue  to 
neglect  much  longer  the  advances  in  economic  theory  and  methods 
pioneered  by  Evans  and  Schultz. 

In  1928  my  father  became  the  Chairman  of  the  Mathematics  Department. 
One  of  his  early  acts  in  this  capacity  was  to  arrange  for  the  loan  by 
the  Bell  Telephone  Laboratories , Inc.  of  a member  of  its  Technical 
Staff,  Dr.  Thornton  C.  Fry,  author  of  Probability  and  Its  Engineering 
Uses  (D.  Van  Nostrand,  1928),  to  give  a course  at  Princeton  on  "Methods 
of  Mathematical  Physics"  as  a Visiting  Lecturer  in  Mathematics  during 
the  first  semester  1929-30.  I remember  going  with  my  father  to  Bell 
Labs  to  visit  Fry  during  either  my  spring  or  summer  vacation  of  1929 — the 
necessary  arrangements  may  have  been  broached,  or  perhaps  firmed  up  on 
that  occasion.  Be  that  as  it  may,  one  result  of  Fry's  visit  to  Princeton 
was  that  a course  in  probability,  taught  by  H,  P.  Robertson  (1903-1964), 
Associate  Professor  of  Mathematical  Physics,  using  Fry's  book  as  the 
text,  was  offered  by  the  Mathematics  Department  during  the  second 
semester  of  my  sophomore  year  (1931-32) . It  was  this  course  that  first 
interested  me  in  probability  and  mathematical  statistics  and  started  me 
on  my  career. 

In  1931  steps  were  taken  that  led  to  a course  in  "modern  statistical 
theory"  being  offered  for  the  first  time  at  Princeton  by  the  Department 
of  Economics  and  Social  Institutions  during  the  first  semester  of  my 
senior  year  (1933-1934).  What  happened  was  this:  Professor  Frank  D. 

Graham  (1890-1949)  of  this  department  approached  my  father  in  his 
capacity  as  Dean  of  the  Faculty,  and  suggested  that  one  way  to  overcome 
Lack  of  competence  In  his  department  with  respect  to  the  latest  develop- 
ments in  mathematical  and  statistical  methods  in  economics  would  be  to 
send  one  of  the  young  instructors  in  his  department  to  study  with 
Professor  Henry  Schultz  at  the  University  of  Chicago,  (The  possibility 
of  hiring  a new  staff  member  from  the  outside  to  this  end  had  been 
considered  earlier  but  put  aside — the  Depression  was  in  full  swing, 
and  there  was  a freeze  on  new  University  appointments.)  My  father  was 
favorable  to  this  proposition,  subject  to  an  additional  provision: 
that  the  individual  concerned  also  study  the  modem  theory  of  statistical 
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inference  with  Harold  Hotelling  for  the  purpose  of  initiating  a course 
in  this  subject  on  his  return.  The  "victim"  that  Professor  Graham  had 
in  mind  was  Acheson  J.  Duncan;  and  this  is  how  it  came  to  pass  that 
Duncan,  with  financial  assistance  from  the  International  Finance 
Section  of  Princeton  University,  spent  the  first  half  of  the  academic 
year  1931-32  studying  with  Professor  Henry  Schultz  at  the  University  of 
Chicago;  and  the  second  half  with  Professor  Hotelling  at  Columbia.-^' 

When  Duncan  arrived  at  Columbia  University  early  in  1932,  one  of 
the  first  persons  he  met  was  Wilks.  Another  was  W.  R.  Pabst,  then  a 
graduate  student  in  Economics  at  Columbia,  who  years  later,  was  to  be 
instrumental  in  Duncan's  becoming  active  as  a teacher,  author,  and 
consultant  on  statistical  methods  in  standardization  and  quality 
control.  Duncan  returned  to  Princeton  in  the  fall  of  1932,  and  began 
to  ready  himself  to  teach  his  projected  new  courses,  unaware — as  were 
also  Wilks  and  my  father — that  before  his  course  in  "modern  statistical 
theory"  would  get  under  way,  Wilks  would  have  joined  the  Princeton 
University  faculty. 

The  program  worked  out  for  Duncan  on  his  return  to  Princeton 
was  this:  He  would  participate  as  an  assistant  in  the  course, 

"Elementary  Statistics",  taught  by  Professor  James  G,  Smith  0-89 7—1946) 
in  the  Department  of  Economics  and  Social  Institutions,  scheduled  for 
the  Spring  semester  in  1933,  serving  as  instructor  in  charge  of  the 
"laboratory"  or  "workshop"  sessions  in  which  the  students  gained  practical 
experience  in  graphical  and  tabular  presentation,  and  in  the  computation 
of  descriptive  statistics,  index  numbers,  moving  averages,  link  relatives, 
etc.  Then,  as  a sequel  to  this  course,  Duncan's  new  course  on  "Modem 
statistical  theory"  would  be  offered  by  the  same  Department  during  the 
first  semester  of  the  academic  year  1933-34. 

I took  these  two  courses  in  the  Spring  and  Fall  of  1933,  respectively. 
In  Smith's  course  we  used  as  text  Principles  and  Methods  of  Statistics 
by  Robert  E.  Chaddock  (1879-1940),  published  by  the  Houghton  Mifflin 
Company  in  1925,  but  the  scope,  nature,  and  mode  of  presentation  is  more 
accurately  reflected  by  Professor  Smith's  Elementary  Statistics.  An 
Introduction  to  the  Principles  of  Scientific  Methods,  published  the 
following  year  (New  York:  Henry  Holt  and  Company,  1934).  Some  of 

R.  A.  Fisher's  contributions  to  statistical  methodology  were  alluded  to, 
but  only  very  briefly,  as  tips  on  recent  developments  that  would  warrant 
looking  into,  not  as  integral  parts  of  the  course.  In  Duncan's  course, 
on  the  other  hand,  built  as  it  was  around  Hotelling's  lectures,  and  the 
then  available  mimeographed  chapters  of  Hotelling's  never  published  book. 
Statistical  Inference,  the  contributions  of  Student  and  R.  A,  Fisher 
occupied  the  center  of  the  stage  a large  part  of  the  time. 
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In  the  spring  of  1933  a crisis  developed  of  which  I was  totally 
unaware  at  the  time,  and  the  particulars  of  which  I was  not  to  learn 
until  some  years  later.  Wilks  was  at  Cambridge  University  working  with 
Wishart  on  the  last  lap  of  his  two-year  fellowship  program  and  would  be 
needing  a permanent  post,  or  at  least  a new  source  of  income,  by  fall. 

He  had  sent  rdsum£s  of  his  professional  career  to  the  universities  in  the 
United  States  known  to  have  programs  in  probability  and  mathematical 
statistics,  indicating  that  he  was  in  need  of  an  instructorship  or  other 
full-time  position  beginning  with  the  academic  year  1933-34.  The  replies 
that  he  received  were  all  negative — the  United  States  was  in  the  depth 
of  the  Depression,  colleges  and  universities  were  having  to  make  do  with 
dramatically  reduced  income  from  endowment  and  other  sources,  and  all, 
it  seemed,  were  tightening  the  belt,  and  none  were  planning  to  take  on 
additional  personnel.  With  an  exceptional  training  in  mathematical 
statistics,  with  four  substantial  research  papers,  and  two  research  notes 
already  published,  one  joint  research  paper  accepted  for  publication, 
and  two  research  papers  nearly  ready  for  publication,  he  was  one  of  the 
most  promising  young  men  in  mathematical  statistics  and  applied 
mathematics  generally,  yet  he  had  no  prospect  of  a job.  Wilks’s 
situation  seemed  hopeless  and  was  rapidly  becoming  desperate.  Here 
he  was  in  England  with  his  wife  and  son;  his  fellowship  funds,  which 
were  never  really  adequate  for  married  people,  or  couples  with  children, 
were  about  to  run  out;  and  no  prospect  of  employment. 

Hotelling,  knowing  full  well  of  my  father’s  desire  to  build  up  a 
program  in  probability  and  mathematical  statistics  at  Princeton  and  of 
the  need  of  the  College  Entrance  Examination  Board  for  assistance  from 
someone  of  Wilks's  caliber  on  multivariate  sampling  distribution  problems 
arising  in  educational  testing,  appealed  directly  to  my  father  to  take 
Wilks  on  at  Princeton,  stressing  the  long-term  advantages  to  Princeton 
and  the  a t-th e-moment  desperateness  of  Wilks’s  situation.  Thus  it  came 
to  pass  late  in  the  spring  of  1933  that  my  father,  as  Chairman  of  the 
Mathematics  Department,  offered  Wilks  an  instructorship  in  the  Department 
of  Mathematics  for  the  academic  year  1933-34,  and  advised  him  of  a 
tentative  arrangement  that  he  had  made  with  Professor  Carl  C.  Brigham  of 
the  Department  of  Psychology  and  Associate  Secretary  of  the  College 
Entrance  Examination  Board  (the  central  office  of  which  had  been  at 
Princeton  for  some  years)  to  work  part-time  also  with  the  Board  on 
problems  arising  in  the  scaling  of  achievement  tests.  It  was  not  until 
many  years  later  that  I learned  from  my  father  that  he  had  brought  off 
this  coup  over  the  opposition  of  almost  every  member  of  his  Department. 

I have  often  wondered  whether  he  would  have  been  able  to  bring  it  off 
a year  or  even  six  months  later  because,  although  he  continued  as  Chairman 
of  the  Mathematics  Department  until  1945,  in  mid-1933  he  gave  up  his 
post  as  Dean  of  the  Faculty  to  become  Dean  of  the  Graduate  School, 
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I also  learned  In  later  years,  after  I had  returned  from  London 
and  had  become  a close  personal  friend  of  Sam  and  Gena  Wilks,  that 
Sam  had  received  only  one  other  offer:  at  Rothamsted  Experimental 

Station,  in  response  to  Wishart's  repeated  pressuring  of  R.  A.  Fisher 
on  Wilks  s plight  and  need.  The  offer  itself,  however,  was  humiliatingly 
^^■88^^-dly  snd  grossly  inadequate  to  Wilks  's  needs,  perhaps  as  a result 
of  Wilks  having  already  incurred  Fisher's  wrath  over  his  analytical 
(in  contrast  to  geometrical)  exposition  of  the  independence  of  sums  of 
squares  in  the  analysis  of  variance. 


Wilks  arrived  in  Princeton  in  September  1933.  As  a new  instructor 
in  the  Department  of  Mathematics,  he  found  himself  teaching  the  usual 
undergraduate  courses  in  analytic  geometry,  calculus,  and  so  forth 
during  the  academic  year  1933-34.  In  addition  to  such  teaching  that 
first  year,  Sam  continued  his  research,  primarily  in  multivariate 
analysis;  gave  me  helpful  guidance  in  the  preparation  on  my  senior 
thesis  on  "The  Accuracy  of  Computations  Involving  Quantities  Known 
Only  to  a Given  Degree  of  Approximation";  and  spent  the  remainder  of 
his  "spare  time"  on  his  "second  job"  with  Professor  Brigham  and  the 
College  Entrance  Examination  Board.  The  following  year,  1934-35, 

Sam’s  program  was  much  the  same,  except  that  he  now  guided  my  post- 
graduate reading  and  study  in  probability  and  statistical  theory  and 
methodology  in  preparation  for  my  becoming  a doctoral  candidate  in 
Statistics  under  J.  Neyman  and  E.  S.  Pearson  at  University  College, 
London,  1935-37. 

Wilks  taught  his  first  statistics  course  at  the  University  of 
Pennsylvania,  in  Philadelphia,  during  1935-36.  (Dr.  George  Galley 
Chambers,  Professor  of  Mathematics,  University  of  Pennsylvania,  had 
died  on  24  October  1935,  shortly  after  his  graduate  course  "Modern 
Theory  of  Statistical  Analysis"  had  gotten  under  way.  Sam  was 
commissioned  to  complete  the  teaching  of  this  course  in  his  stead.) 
During  the  same  period  Sam  gave  an  Informal  course — i.e.,  not  listed 
in  the  official  University  course  catalog — to  three  Princeton  seniors, 
Walter  W.  Merrill,  John  0.  Rohm,  and  William  C.  Shelton,  on  much  the 
same  material;  and  supervised  Shelton's  senior  thesis  on  "Regression 
and  Analysis  of  Variance".  (Shelton  continued  in  Statistics,  rising 
to  become  Special  Assistant  to  the  Commissioner  of  Labor  Statistics. 
Merrill  and  Rohm  took  up  accounting  and  law,  respectively.) 

Wilks  was  promoted  to  an  assistant  professorship  in  1936;  and 
in  1936-37  taught  his  first  statistics  courses  at  Princeton:  a 

graduate  course  during  the  Fall  Term — see  WILKS  1937 — and  an  under- 
graduate course  during  the  Spring  Term.  A Princeton  senior  that  year 
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who  took  the  graduate  course,  Irving  E.  Segal  (now  a Prof,  of  Math 
at  MIT),  wrote  a senior  thesis  under  Sam's  supervision  that  was 
subsequently  published  in  the  Proceedings  of  the  Cambridge  Philosophical 
Society  (SEGAL  1938). 

The  publication,  in  the  January  1973  issue  of  the  IMS  Bulletin, 
of  Professor  Harry  C.  Carver's  letter  of  14  April  1972  to  Professor 
William  Jackson  Hall  on  the  "beginnings  of  the  Annals " prompts  me 
to  correct  a mistaken  conjecture  contained  therein  on  why  Sam  Wilks 
was  not  permitted  to  teach  a course  in  mathematical  statistics  during 
his  first  few  years  as  an  instructor  in  the  Mathematics  Department 
there.  Professor  Carver  wrote: 

"...one  day  I asked  [Wilks]  how  it  was  that  he  was  not 
teaching  a course  in  mathematical  statistics  at  Princeton. 

He  replied  that  he  had  tried  to  start  such  a course  there, 
but  his  superiors  turned  down  his  request  each  time, — 
probably  because  mathematical  statistics  and  probability 
had  not  yet  rung  a bell  in  the  staid  Eastern  Colleges." 

The  fact  of  the  matter  is  that  mathematical  statistics  and 
probability  already  had  "rung  a bell"  at  Princeton:  two  years  before 

Wilks's  arrival,  Acheson  J.  Duncan  had  been  sent  off  at  University 
expense  to  study  with  Professors  Henry  Schultz  and  Harold  Hotelling 
for  the  express  purpose  of  readying  himself  to  initiate  courses  in 
"mathematical  economics"  and  "modern  statistical  theory"  on  his  return. 

It  was  this  prior  arrangement  and  commitment,  not  lack  of  appreciation 
of  the  importance  of  mathematical  statistics  and  probability — or  of 
Wilks's  exceptional  qualifications — that  constituted  the  primary 
obstacle  to  Wilks’s  offering  an  undergraduate  course  in  mathematical 
statistics  during  his  first  three  years  as  a member  of  the  Mathematics 
Department  of  Princeton  University.  Duncan's  course  on  "modem  statistical 
theory"  had  been  scheduled  to  be  offered  for  the  first  time  during  the 
Fall  Term  of  1933  before  the  possibility  of  Wilks’s  coming  to  Princeton 
had  even  been  considered.  In  view  of  the  expense  that  the  University 
had  incurred  in  underwriting  Duncan’s  year  of  training  in  preparation 
for  the  offering  of  this  course,  and  the  sacrifice  that  Duncan  had 
made  in  postponing  work  on  his  doctoral  dissertation  in  order  to 
acquire  the  requisite  training  at  the  University's  request,  it  would 
have  been  very  improper  and  cruel  to  have  shelved  Duncan's  course  and 
let  Wilks  start  one  instead.  I am  sure  that  Wilks  recognized  this;  and 
was  also  cognizant  of  the  other  factors  that  delayed  his  getting  a 
course  of  his  own  in  the  Mathematics  Department. 

The  three-year  delay  between  Sam's  arrival  at  Princeton  and 
his  first  officially  recognized  course  in  statistics  under  the  auspices 
of  the  Mathematics  Department  was  the  result  of  at  least  four  factors. 
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First,  there  was  the  priority  that  circumstances  had  accorded  to 
Duncan’s  course  in  the  Department  of  Economics  and  Social  Institutions. 
Furthermore,  that  Department  had  taken  the  initiative  In  the  matter, 
and  was  desirous  of  modernizing  its  outlook  and  course  offerings  with 
respect  to  mathematical  economics  and  statistics.-^' 

Second,  under  the  circumstances,  any  course  on  ’mathematical 
statistics”,  "statistical  analysis",  "statistical  inference",  or 
whatever,  to  be  offered  by  Wilks  in  the  Mathematics  Department  would 
have  to  be  an  additional  new  course,  and  would  require  the  approval 
of  the  all-powerful  Course  of  Study  Committee  of  the  Faculty.  A new 
course  at  Princeton  had  to  be  described  in  detail  by  the  department 
proposing  to  offer  it.  Faculty  approval  gave  the  department  the  right 
to  teach  the  described  subject  matter.  I am  not  sure  that  this  was 
an  exclusive  right,  but  I doubt  that  the  Course  of  Study  Committee 
would  have  approved  teaching  essentially  the  same  material  in  two 
departments.  Hence  a major  obstacle  to  Sam's  teaching  an  undergraduate 
course  in  Statistics  was  the  historical  fact  that  Statistics  had  been 
the  province  of  the  Department  of  Economics  and  Social  Institutions. 

Third,  until  Sam  was  promoted  to  an  assistant  professorship  in 
1936,  he  was  only  an  instructor;  and  in  a department  having  the  stature, 
nationally  and  internationally,  of  Princeton's  Mathematics  Department 
it  was  definitely  not  customary  for  an  undergraduate,  much  less  a 
graduate  course,  to  be  initiated  by  and  be  the  sole  responsibility  of 
an  individual  with  the  rank  of  instructor. 

A fourth,  and  very  inhibiting  factor  was  the  unfavorable  mathematical 
"climate"  that  prevailed  in  Fine  Hall,  which  housed  Princeton's  Mathe- 
matics Department  during  Sam's  early  years  at  Princeton.  Geometry  had 
occupied  the  center  of  the  stage  in  this  Department,  for  over  a quarter 
of  a century,  with  Algebra  and  Analysis  accorded  much  less  exalted 
roles.  Then,  in  1932,  the  new  Institute  for  Advanced  Study,  an 
institution  completely  distinct  from  Princeton  University,  had  come  into 
being,  and  the  members  of  its  School  of  Mathematics  were  granted  office 
space  in  the  Mathematics  Department’s  Fine  Hall  until  the  completion  of 
their  first  building,  Fuld  Hall,  in  1939.  Albert  Einstein  (1879-1955) 
arrived  to  take  up  his  post  in  the  Institute  during  the  Winter  of  1933, 
and  Hermann  Weyl  (1885-1955)  arrived  a few  months  earlier.  John  Von 
Neumann  (1903-1957)  was  already  there  (Lecturer,  1930-31 , Princeton,  then  Professor 
of  Mathematical  Physics,  1931-33;  Professor  of  Mathematics,  Institute 
for  Advanced  Study,  1933-57);  as  were  also  E.  U.  Condon  (1903-1961; 

Assistant  Professor  of  Mathematical  Physics,  Princeton,  1928-31; 

Associate  Professor,  1931-38,  Professor,  1938-47),  and  E.  P.  Wigner 
(Lecturer  in  Mathematical  Physics,  Princeton,  1930;  Professor,  1930-36; 

1938-1971).  With  this  galaxy  of  mathematical  physicists  all  together 
ine  one  place  for  the  first  time,  the  mathematical  theory  of  relativity 
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and  quantum  mechanics  were  definitely  the  fashion  of  the  day  in  Fine 
Hall-* 'a  difficult  "climate'1  in  which  to  initiate  a program  in 
mathematical  statistics. 

By  1936-37,  the  division  of  territory  between  the  Department  of 
Mathematics  and  the  Department  of  Economics  and  Social  Institutions 
had  been  resolved.  The  latter  would  be  restricted  to  Instruction  in 
statistical  theory  and  methods  pertinent  to  the  economic  and  social 
sciences;  and  the  basic  general  undergraduate  course (s)  in  statistical 
theory  and  methodology,  and  the  graduate  courses  in  advanced  mathematical 
statistics  would  be  the  province  of  the  Mathematics  Department.  As  we 
have  already  said,  Wilks  taught  his  first  statistics  course  at  Princeton 
in  the  fall  of  1936,  the  graduate  course  leading  to  his  lithographed 
lecture  notes  on  Statistical  Inference — (1937)  ; and  in  the  spring  of 
1937,  a sophmore  course  with  calculus  as  prerequisite,  quite 
possibly  the  first  carefully  formulated  college  underclass  course 
:Ln  mathematical  statistics  at  this  level.  It  was  offered  thereafter 
for  a number  of  years  to  students  in  all  fields  in  the  second  half  of 
the  sophomore  year.  The  material  presented  in  this  course,  extended 
and  polished,  became  generally  available  a decade  later  in  his  "blue 
book".  Elementary  Statistical  Analysis  (1948b)  . A third  course,  also 
one  semester  in  length,  was  added  in  1939-40.  It  was  an  upperclass 
course  for  students  who  wanted  to  specialize  in  statistics,  and 
consisted  of  a rather  thorough  mathematical  treatment  of  statistical 
theory  in  the  classroom  plus  a laboratory  section  devoted  to  applica- 
tions and  computations . This  course  was  taken  also  by  beginning  graduate 
students.  Wilks's  first  doctoral  student,  Joseph  F.  Daly  received 
his  Ph.D.  in  1939.  George  W.  Brown  and  Alexander  M.  Mood  followed  in 
1940.  World  War  II  demolished  his  plans  for  sabbatical  leave  to  lecture 
in  South  America  and  accept  an  offered  exchange  professorship  for  one 
semester  at  the  National  University  in  Santiago,  Chile.  As  World  War  II 
progressed,  Sam  became  ever  more  deeply  involved  in  war  research — 1 
shall  return  to  this  in  a moment — and  in  due  course  was  released  from 
academic  duties  entirely.  Helped  by  two  of  his  graduate  students, 

T.  W.  Anderson  and  D.  F.  Votaw,  Jr.,  and  Henry  Scheffd,  he  succeeded  in 
seeing  through  to  lithoprinted  publication  the  graduate  level  text. 
Mathematical  Statistics  (1943),  before  becoming  totally  involved  in 
war  work.  This  was  the  forerunner  of  his  polished  comprehensive  treat- 
ment bearing  the  same  title  published  as  a type-set  book  in  1962. 

In  keeping  with  my  father's  policy  of  promotions  as  soon  as  merited 
without  regard  to  leave  of  absence,  Sam  was  promoted  to  a full  Professor 
of  Mathematics  in  1944,  effective  on  his  return  to  academic  duties; 
and  plans  were  laid  for  a Section  of  Mathematical  Statistics  within 
the  Department  of  Mathematics.  Following  the  war  there  was  a steady 
flow  of  able  graduate  students  and  postdoctoral  research  associates, 
some  of  whom,  like  Robert  Hooke  and  Henry  Scheffd,  were  changing  from 


mathematics  to  statistics.  By  the  time  of  Sam’s  death  (1964),  Princeton 
had  granted  Ph.D.'s  to  approximately  40  men  in  mathematical  statistics  and 
probability,  all  of  whom  had  studied  to  some  extent  with  Wilks,  and  the 
dissertations  of  about  half  had  been  supervised  by  him. 

It  would  be  a mistake  to  infer  from  the  foregoing  that  Wilks  ’s 
educational  activities  were  limited  to  teaching  and  thesis  guidance  in 
mathematical  statistics.  He  was  deeply  interested  in  the  whole  spectrum 
of  mathematical  education.  In  "Personnel  and  Training  Problems  in 
Statistics"  (1947)  he  outlined  the  growing  use  of  statistical  methods, 
the  demand  for  personnel,  problems  of  training,  and  made  recommendations 
that  served  as  a guide  in  the  rapid  growth  of  university  centers  of 
training  in  statistics  after  World  War  II.  Drawing  on  his  experience 
at  Princeton,  he  urged,  in  "Teaching  Statistical  Inference  in  Elementary 
Mathematics  Courses"  (1958),  teaching  the  principles  of  statistical 
inference  to  freshman  and  sophmores , and  further  proposed  revamping 
high  school  curricula  in  mathematics  and  the  sciences  to  provide  topics 
In  probability,  statistics,  logic  and  other  modern  mathematical  subjects. 

In  furtherance  of  his  ideas  in  this  direction  he  co-authored,  as  a 
member  of  the  Commission  on  Mathematics  of  the  College  Entrance  Examina- 
tion Board  1955-1958,  the  Introductory. . . Exp e r imen t al  Course  (1957)  that 
recommended  major  changes  in  the  teaching  of  mathematics  in  the  secondary 
schools  and  suggested  inclusion  of  an  option  of  Introductory  Probability 
with  statistical  applications  in  the  twelfth  grade.  During  his  last 
few  years  he  worked  with  an  experimental  program  in  Miss  Mason's  School 
In  Princeton  which  introduced  new  mathematics  at  the  elementary  level, 
down  to  kindergarden . During  his  final  week  of  life,  he  was  considering, 
as  a member  of  the  Advisory  Board  of  the  School  Mathematics  Study  Group, 
how  much  time  the  following  summer  he  would  be  able  to  devote  to  writing 
on  probability  and  statistics  for  this  group. 

5.  WILKS’S  FURTHER  CONTRIBUTIONS  TO  MATHEMATICAL  STATISTICS.  A 
few  more  words  are  in  order  on  Wilks's  further  contributions  to  mathematical 
statistics  before  turning  to  his  many  services  to  the  U.S.  Government 
generally  and  to  the  Army  in  particular. 

Wilks  was  definitely  not  an  ivory  tower  researcher.  A great  many 
of  his  research  papers  in  mathematical  statistics  were  written  to  meet 
needs  that  he  personally  had  encountered  in  his  applied  work;  and, 
especially  in  his  earlier  papers,  he  usually  included  explicit  worked 
examples  of  the  application  of  the  new  theory  concerned.  Thus,  his 
first  Important  contribution  to  multivariate  analysis  after  arriving 
in  Princeton,  "On  the  Independence  of  k.  Sets  of  Normally  Distributed... 
Variables"  (1935a),  appears  to  have  been  written  to  meet  a need  Wilks 
encountered  in  his  work  with  the  College  Entrance  Examination  Board  in 
Princeton,  N.J.;  as  do  also  many  of  his  later  contributions  to  multi- 
variate analysis,  e.g,,  "Weighting  Systems  for  Linear  Functions  of 
Correlated  Variables.#  •"  (1938b)and  "Sample  Criteria  for  Testing  Equality 
of  Means,  Equality  of  Variance,  and  Equality  of  Covariances ... " (1946); 
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and  "Multivariate  Statistical  Outliers"  (1963),  the  last  of  his  total 
of  fifteen  research  papers  on  topics  in  multivariate  analysis,  has  a 
definitely  applied  flavor. 

In  addition  to  the  extensive  and  penetrating  studies  of  likelihood 
ratio  tests  for  various  hypotheses  relating  to  multivariate  normal 
distributions  embodied  in  the  aforementioned  papers,  Wilks  investigated 
(1935b)  likelihood  ratio  tests  for  various  hypotheses  relating  to 
multinomial  distributions  and  to  independence  in  two-  three-  and  higher- 
dimensional contingency  tables,  and  provided  (1938a)  a compact  proof 
of  the  basic  theorem  on  the  large-sample  distribution  of  the  likelihood 
ratio  criterion  for  testing  "composite"  statistical  hypotheses,  i.e., 
when  the  "null  hypothesis"  tested  specifies  the  values  of,  say,  only 
m out  of  the  h parameters  of  the  probability  distribution  concerned. 

Jerzy  Neyman's  basic  paper  on  the  theory  confidence-interval  estimation 
appeared  in  1937.  The  following  year  Wilks  showed  (1938c)  that,  under 
fairly  general  conditions,  confidence  intervals  for  a parameter  of  a 
probability  distribution  based  upon  its  maximum- likelihood  estimator 
are  on  the  average  the  shortest  obtainable  in  large  samples;  and  a year 
later,  in  a joint  paper  with  J.  F.  Daly,  generalized  this  result  to  the 
case  of  several  parameters. 

In  response  to  a need  expressed  by  Shewhart,  Wilks,  in  "Deter- 
mination of  Sample  Sizes  for  Setting  Tolerance  Limits"  (1941) , laid 
the  foundations  of  the  theory  of  statistical  "tolerance  limits", 
which  are  actually  confidence  limits,  in  the  sense  of  Neyman’s  theory, 
not,  however,  for  the  value  of  some  parameter  of  the  distribution 
sampled  as  in  Neyman's  development,  but  rather  for  the  location  of 
a specified  fraction  of  the  distribution  sampled.  In  this  paper  he 
showed  that  a suitably  selected  pair  of  ordered  observations  ("order 
statistics")  in  a sample  of  sufficient  size  from  an  arbitrary  continuous 
distribution  provide  a pair  of  limits,  statistical  "tolerance  limits", 
to  which  there  corresponds  a stated  chance  that  at  least  a specified 
fraction  of  the  underlying  distribution  is  contained  between  these 
limits,  thus  providing  the  "distribution-free"  solution  needed  when  the 
assumption  of  an  underlying  normal  distribution  of  industrial  production 
is  unwarranted.  In  the  same  paper  he  derived  the  corresponding  parametric 
solution  of  maximum  efficiency  in  the  case  of  sampling  from  a normal 
distribution  (based  on  the  sample  mean  and  standard  deviation) , and  an 
expression  for  the  relative  efficiency  of  the  distribution-free  solution 
in  this  case.  In  "Statistical  Prediction..."  (1942),  he  found  formulas 
for  the  probabilities  that  at  least  a fraction  NQ/N  of  a second  random 
sample  of  N observations  from  an  arbitrary  continuous  distribution 
would  (a)  lie  above  the  "order  statistic"  (rth  observation  in 

increasing  order  of  size),  1 £ r £ n,  in  a first  random  sample  of  size  n 
from  the  same  distribution;  (b)  be  included  between  the  rth  and  sth 
order  statistics,  1 £ r £ s £ n,  of  the  first  sample;  and  illustrated 
the  application  of  these  results  to  the  setting  of  one-  and  two-sided 
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statistical  tolerance  limits.  These  papers  embodied  the  earliest  of  a 
series  of  contributions  made  by  Wilks  to  "nonparametric"  or  "distribution- 
free"  methods  of  statistical  inference,  an  area  of  research  in  which  he 
persuaded  a number  of  his  students  to  write  senior  theses  or  doctoral 
dissertations;  and  of  which  he  provided  an  extensive  review  in  depth  in 
"Order  Statistics"  (1948a),  an  expository  paper  that  was  in  large  part 
responsible  for  the  ensuing  blossoming  of  research  activity  in  this  area. 

Wilks  was  one  of  the  small  group  of  mathematicians  and  statisticians 
who  at  Ann  Arbor,  Michigan,  on  September  12,  1935,  founded  the  Institute 
of  Mathematical  Statistics,  and  thereafter  was  an  active  and  leading 
member.  At  this  meeting,  Harry  C.  Carver,  who  had  founded,  edited,  and 
personally  financed  and  published  the  Annals  of  Mathematical  Statistics 
(in  affiliation  with  the  American  Statistical  Association)  from  1930, 
volunteered  to  turn  over  the  editing  and  publication  of  Annals  to  the 
Institute  as  its  official  organ  as  soon  as  the  Institute  was  able  to 
assume  these  responsibilities.  The  Institute  assumed  full  responsibility 
for  the  Annals , and  Wilks  took  over  as  editor,  with  the  June  1938  issue."' 
He  served  as  editor  through  the  December  1949  issue,  and  guided  the 
development  of  the  Annals  from  a marginal  journal  with  a small  subscription 
list,  to  the  foremost  publication  in  its  field,  with  a ten-fold  increase 
in  individual,  and  a five-fold  Increase  in  library  subscriptions;  and 
in  the  process,  fostered  the  growth  of  the  Institute,  from  a once 
marginal  society  to  a mature  international  organization,  large  in  both 
size  and  contribution.  His  editorship  of  the  Annals  was  his  greatest 
contribution  to  mathematical  statistics. 

In  1954  Wilks  joined  Walter  Shewhart  in  editing  the  Wiley  Publications 
in  Statistics,  a major  U.S.  publication  effort  that  did  much  to  change 
statistics  from  a subordinate  branch  of  the  social  sciences  in  the  1930’s, 
to  a respected  discipline  in  its  own  right  with  a large  and  solid 
literature  in  the  1960’s, 

6.  HIS  BROAD  CAREER  OF  GOVERNMENT  SERVICE,  AND  AS  INITIATOR  OF 
THESE  EXPERIMENT  DESIGN  CONFERENCES.  In  1936,  when  my  father  recommended 
Sam  for  promotion  to  Assistant  Professor  of  Mathematics  he  noted  in  his 
recommendation  that  Sam  had  Just  received  an  appointment  as  a Collaborator 
in  a United  States  Soil  Conservation  Program  of  the  Department  of 
Agriculture.  A broad  career  of  government  service  was  underway  that 
was  to  range  widely  and  continue  through  the  last  twenty-eight  years  of 
his  life.  He  served  the  United  States  Government  as  a member  of  the 
Applied  Mathematics  Panel,  NDRC,  OSRD,  and  director  of  its  Princeton 
Statistical  Research  Group,  1942-1945;  chairman,  mathematics  panel, 

Research  and  Development  Board,  DOD,  1948-1950;  member,  scientific 
advisory  committee,  Selective  Service  System,  1948-1953;  "charter" 
member,  ASA  advisory  committee  to  the  Bureau  of  the  Budget,  1951-1964; 
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member,  divisional  committee  for  the  mathematical,  physical  and  engineering 
sciences,  NSF,  1952-1956;  member,  committee  on  battery  additives,  NAS, 

1953;  member,  divisional  committee  for  the  social  sciences,  NSF,  1957-1962; 
member,  scientific  advisory  board,  NSA,  1953-1964  (chairman,  1958-1960); 
member,  U.S.  National  Commission  for  UNESCO,  1960-1962;  and  academic 
member.  Army  Mathematics  Advisory  Panel  (called  "Army  Mathematics 
Steering  Committee",  from  1956  on),  1954-1964,  It  was  in  this  latter 
capacity  that  he  initiated  these  Experiment  Design  Conferences 

General  Leslie  E.  Simon,  upon  becoming  Chief  of  the  Research  and 
Development  Division  in  the  Office,  Chief  of  Ordnance,  in  1951,  entered 
into  an  agreement  with  Duke  University  to  establish  on  that  campus,  an 
Office  of  Ordnance  Research  to  sponsor  external  basic  research  initiated 
by  non -government  investigators  with  ordnance  interests.  Such  research 
had  always  been  carried  out  by  all  Army  Technical  Services,  but  previously 
under  vague  mandate  and  seldom  on  an  appreciable  scale.  The  level  of 
effort  had  been  wholly  dependent  on  the  sophistication  of  the  administrators 
concerned.  A Statistics  Branch,  and  other  units  with  statistical  interests, 
were  included  in  the  setup. 

In  1954  the  Army  Research  Office — Durham  (then  the  Office  of  Ordnance 
Research)  upon  the  request  of  the  Chief  of  Research  and  Development 
Division,  Office,  Assistant  Chief  of  Staff  G-4,  Department  of  the  Army, 
established  the  Army  Mathematics  Advisory  Panel  (AMAP)  as  an  ad  hoc 
committee  to  provide  advice  on  the  mathematical  needs  of  the  Army,  (The 
Panel  was  reconstituted  as  a permanent  body,  the  Army  Mathematics  Steering 
Committee,  on  27  February  1956.) 

Soon  after  its  formation,  the  AMAP  conducted  a comprehensive  inquiry 
into  the  Army’s  uses  of  mathematics;  whether  these  uses  could  be 
advantageously  extended;  what  future  needs  might  be  anticipated;  and  what 
measures  might  then  be  taken  to  insure  a future  capability  adequate  to 
these  needs.  As  an  academic  member,  Wilks  surveyed  thirty  Army  installations 
with  the  AMAP  and  reported  that  ’’the  most  frequently  mentioned  needs 
expressed  by  the  scientific  personnel  were  for  greater  knowledge  of 
modem  statistical  theory  of  the  design  and  analysis  of  experiments" 

(SIMON  1965,  p.  958),  clearly  implying  that  a major  deficiency  of  Army 
research,  development  and  testing  was  insufficient  use  of  modern 
statistical  experiment  design  techniques.  He  proposed,  therefore,  that 
the  Army  establish  a series  of  Army-wide  conferences  on  design  of 
experiments  in  Army  research,  development  and  testing.  Dr.  Frank  E. 

Grubbs,  who  had  chaired  an  Ordnance  symposium  on  Statistical  Methods  in 
1953,^'  strongly  indorsed  Wilks’  proposal  for  Army-wide  conferences 
devoted  primarily  to  design  of  experiments.  General  Simon  gave  the 
proposal  a green  light  and  his  support . Upon  making  further  inquiries 
it  was  found  that  a number  of  research  workers  at  various  facilities 
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expressed  an  interest  in  contributing  papers  to  such  a conference. 

Others  had  unsolved  or  partially  solved  problems  which  they  wished  to 
present  for  discussion. 

The  AMAP  decided  to  organize  a three-day  conference  on  the  design 
of  experiments  with  three  kinds  of  sessions.  The  first  group  of  sessions 
would  consist  of  invited  papers  by  well-known  authorities  on  the  philosophy 
and  general  principles  of  the  design  of  experiments.  The  second  group 
would  consist  of  technical  papers  contributed  by  research  workers  from 
various  Army  research,  development  and  testing  facilities.  The  third 
group  would  be  clinical  sessions  consisting  of  presentations  and 
discussions  of  partially  solved  and  unsolved  problems  which  had  arisen 
in  these  establishments. 

Wilks  agreed  to  serve  as  chairman  of  the  first  Conference,  which  was 
held  on  October  19-21,  1955  at  the  Diamond  Ordnance  Fuze  Laboratories 
and  the  National  Bureau  of  Standards  in  Washington,  D.C.  It  was  attended 
by  over  230  registrants  and  participants  representing  some  50  organizations. 
Speakers  and  other  participants  in  the  conference  came  from  the  Bell 
Telephone  Laboratories,  Johns  Hopkins  University,  Princeton  University, 
Virginia  Polytechnic  Institute,  Bureau  of  Ships,  National  Bureau  of 
Standards,  and  18  Army  facilities 

More  specifically,  the  principal  speakers,  and  their  topics,  were: 

1.  W.  G.  Cochran,  The  Philosophy  Underlying  the  Design  of 
Experiments . 

2.  Churchill  Eisenhart,  The  Principle  of  Randomization  in  the 
Design  of  Experiments . 

3.  M.  E.  Terry,  Finding  Optimum  Conditions  by  Experimentation. 

4.  Panel  Discussion  led  by  John  W.  Tukey  on  How  and  Where  Do 

Statisticians  Fit  In.  (The  others  on  this  Panel  were:  Besse  B. 

Day,  Cuthbert  Daniel,  Churchill  Eisenhart,  M.  E.  Terry,  and 

S.  S.  Wilks). 

5.  W.  J.  Youden,  Design  of  Experiments  in  Industrial  Research 
and  Development. 

It  was  such  a success  that  the  Army  has  continued  these  conferences 
annually  in  October  or  November  since  1955,  following  the  same  format. 

(See  the  Appendix  for  places  and  dates  of  the  first  nineteen  Conferences, 
and  names  and  topics  of  the  invited  speakers  at  these  Conferences.) 

Wilks  chaired  the  first  nine  of  these  Conferences  (1955-1963),  and  wrote 
the  Foreword  to  the  Proceedings  of  the  first  eight.  At  the  tenth 
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Conference,  held  in  1964  and  dedicated  to  Wilks's  memory,  establishment 
of  the  Samuel  S.  Wilks  Memorial  Award  and  Medal  was  announced,  to  be 
administered  by  the  American  Statistical  Association,  and  to  be  awarded 
annually  "to  a statistician. . .based  primarily  on  his  contributions ...  to 
the  advancement  of  scientific  or  technical  knowledge  in  Army  statistics, 
ingenious  application  of  such  knowledge,  or  successful  activity  in  the 
fostering  of  cooperative  matters  which  coincidentally  benefit  the  Army, 
the  DOD  and  the  Government,  as  did  Samuel  S.  Wilks  himself";  and  the 
initial  award  presented  to  Dr.  Frank  E.  Grubbs,  Ballistic  Research 
Laboratories,  Aberdeen  Proving  Ground.  In  1947,  Wilks  was  awarded 
the  Presidential  Certificate  of  Merit  for  his  contributions  toward 
antisubmarine  warfare  and  the  solution  of  convoy  problems ; and  the  same 
year,  the  Centennial  Alumni  Award  of  the  University  of  Iowa. 

6.  HIS  DEATH,  AND  CONCLUDING  REMARKS. 

Sam  became  "my  teacher"  and  guiding  spirit  at  once  in  1933;  and 
in  later  years,  he  proved  "a  friend  indeed",  on  a number  of  "difficult" 
occasions.  He  died  most  unexpectedly  in  his  sleep,  on  March  7,  1964, 
at  his  home  in  Princeton,  New  Jersey.  At  that  Instant  Statistics 
lost  one  of  its  greatest  champions;  government  agencies,  professional 
societies,  and  the  field  of  education  a devoted  work  mate,  helping 
hand,  and  guide;  and  I,  ,'my  teacher"  and  "a  friend  indeed". 

As  W.  G.  Cochran  has  said:  "He  will  be  long  remembered  with 
affection  and  gratitude:  no  man  of  his  generation  did  as  much  to 

ensure  that  the  rapid  growth  of  statistical  theory,  applications,  and 
education  in  the  United  States  took  place  along  sound  and  healthy  lines." 
(COCHRAN  1964,  p.  191);  and  Egon  S.  Pearson:  "...it  is  hard  to  think 

of  any  mathematical  statistician  of  the  past  30  years  who  combined  to 
a greater  extent  an  excellence  in  the  field  of  theory  with  a power  of 
inspiring  confidence  in  government  agencies,  national  research  institutions, 
and  educational  authorities,  as  a wise  counsellor  in  practical  affairs." 
(PEARSON  1964,  p.  597) 

He  is  survived  by  his  widow,  Gena  Orr  Wilks,  his  son,  Stanley  N. 

Wilks;  a brother,  William  Weldon  Wilks,  three  granddaughters,  one  grand- 
son; and  a host  of  friends. 

8.  POSTSCRIPT  AND  ACKNOWLEDGMENTS.  At  the  Tenth  Conference  (1964) 
dedicated  to  the  memory  of  Professor  Wilks,  I spoke  from  notes  on  "Sam 
Wilks  as  1 Remember  Him".  The  material  presented  was  for  the  most  part 
subsequently  written  up  and  a typescript  prepared,  but  unfortunately  not 
Ln  time  for  publication  in  the  Proceedings  of  that  Conference — nor  in  the 
Proceedings  of  the  Eleventh  Conference,  as  was  suggested.  Portions  of 
the  typescript  were  submitted  to,  and  comments  received  ln  writing  from 
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Alva  E.  Brandt,  Acheson  J.  Duncan,  the  late  Frederick  F.  Stephan  (1903-1971) 
and  the  late  George  W.  Snedecor  (1881-1974).  Large  portions  of  that  pre- 
vious manuscript  have  been  taken  over  bodily  and  incorporated  in  the 
present  text,  with  revisions  in  the  light  of  the  comments  received  from 
the  foregoing,  for  which  I am  very  grateful.  Use  has  also  been  made  of 
comments  received  from  Frederick  Mos teller  on  the  penultimate  draft  of 
a biography  of  Wilks  prepared  for  publication  in  a forthcoming  volume  of 
the  Dictionary  of  Scientific  Biography  (New  York:  Charles  Scribner's 

Sons,  Publishers,  1970-  ),  likewise  gratefully  acknowledged.  In  addition, 

I have  taken  advantage  of,  and  have  very  probably  incorporated  more  than 
I realize,  from  the  obituaries  and  other  memorial  articles  on  Wilks  that 
have  appeared  during  the  past  decade,  especially:  ANDERSON  (1965), 

COCHRAN  (1964),  DIXON  (1965),  HANSEN  (1965a,  1965b),  MOOD  (1965), 

MOSTELLER  (1964,  1968),  PEARSON  (1964),  SIMON  (1965),  STEPHAN  AND  TUKEY 
(1965),  and  TUKEY  (1965).  My  thanks  to  these  for  what  I have  "borrowed", 
explicitly  or  otherwise.  For  whatever  faults  of  commission  or  omission 
still  afflict  this  memorial  to  Sam  Wilks,  I must  assume  full  responsibility. 


NOTES 


1.  Alexander  Me  Far lane  Mood  was  the  second  of  Sam’s  graduate  students 
to  receive  a Ph.D,  (1940)  in  mathematical  statistics  from  Princeton 
University,  After  teaching  at  the  University  of  Texas,  and  serving 
as  a statistician  in  the  Bureau  of  Labor  Statistics,  Mood  returned 
to  Princeton  during  World  War  II  as  a research  associate  in  the 
Statistical  Research  Group-Prlnceton,  engaged  in  war  research  under 
Wilks's  direction  as  an  arm  of  the  Applied  Mathematics  Panel  (AMP) 
of  the  National  Defense  Research  Committee  (NDRC)  of  the  Office  of 
Scientific  Research  and  Development  (OSRD) , under  a contract  between 
Princeton  University  and  the  OSRD.  It  was  as  a member  of  this  group 
that  he  and  Wilfrid  J.  Dixon  wrote  their  famous  memorandum,  later 
published  as  an  article  in  the  Journal  of  the  American  Statistical 
Association  (Vol.  43,  No.  (March  1948),  109-126),  on  the  statistical 
theory  of  the  ’‘up-and-down"  or  "Bruceton"  method  of  obtaining  and 
analyzing  sensitivity  data,  with  which  they  had  become  acquainted  in 
1943  at  the  NDRC’s  Explosives  Research  Laboratory  (now  a unit  of  the 
Bureau  of  Mines,  U.S.  Department  of  the  Interior),  at  Bruceton, 
Pennsylvania.  Subsequently  Mood  became  a professor  of  mathematical 
statistics  at  Iowa  State  College;  deputy  chief,  mathematics  division, 
RAND  Corporation;  president,  General  Analysis  Corporation;  a vice 
president  of  CEIR,  Inc.;  and  at  the  time  of  writing  his  tribute  to 
Wilks,  was  Assistant  Commissioner  of  Education,  U.S.  Office  of 
Education . 

2.  Dodd  had  joined  the  staff  of  the  University  of  Texas  in  1907  as 
Instructor  in  Pure  Mathematics.  He  seems  to  have  been  silent 
publication-wise  until  1912  when  two  papers  by  him  appeared,  one 
on  plane  and  skew  curves,  and  the  other  on  the  method  of  least 
squares  and  orthogonal  transformations.  These  were  followed 
immediately  In  1913  by  four  papers  on  statistical  properties  of  the 
arithmetic  mean,  the  median  and  "other  functions  of  measurements". 

One  of  these  latter,  entitled  "The  probability  of  the  arithmetic 

mean  compared  with  that  of  certain  other  functions  of  the  measurements", 
was  published  in  the  Annals  of  Mathematics  (Vol.  14,  pp.  186-198, 

June  1913),  of  which  my  father  (Luther  Pfahler  Eisenhart,  1876-1965) 
was  then  an  editor.  My  father  seems  to  have  corresponded  with 
Professor  Dodd  with  regard  to  this  paper.  Thereafter  Professor  Dodd 
sent  my  father  reprints  of  many  of  his  subsequent  papers  on  functional 
and  statistical  properties  of  various  types  of  "means".  These  reprints 
proved  to  be  very  helpful  to  me  when  1 became  Interested  in  such 
matters  in  the  early  ’30’s.  I had  the  good  fortune  to  meet  Professor 
Dodd,  when  I went  with  Sam  to  the  Joint  Meeting  of  the  American 
Mathematical  Society  and  Institute  of  Mathematical  Statistics  in 
Indianapolis  in  December  1937.  (For  additional  information  on  Dodd, 
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see  footnote  4;  J.  C.  Poggendorff,  Biographisch-Literarisches 
Handworterbuch  fur  Mathematik. . . , Vol.  5 (1904-1922),  Leipzig  and 
Berlin,  1926,  p.  299;  and  C.  D.  Simmons,  "Edward  Lewis  Dodd, 
1875-1943",  Journal  of  the  American  Statistical  Association  Vol.  38, 
No.  222  (June  1943),  247-248.) 

3,  The  University  of  Iowa  in  Iowa  City  (now  known  as  the  "State 
University  of  Iowa")  was,  in  the  1920 fs,  the  leading  center  in 
the  United  States  for  research  and  training  in  mathematical 
statistics.  It  should  not  be  confused  with  Iowa  State  College  at 
Ames  (renamed  "Iowa  State  University"  on  the  occasion  of  its  centenary 
in  1958),  which,  during  the  same  period,  was  the  leading  center  for 
application  of,  and  teaching  the  application  of,  modem  statistical 
methods  in  the  experimental  sciences,  especially  in  agricultural 
research  and  closely  related  fields. 

4.  Professor  Dodd  after  receiving  his  Ph.D.  in  mathematics  from  Yale 
in  1904,  had  served  as  an  Instructor  in  mathematics  for  two  years 
(1904-06)  at  the  University  of  Iowa,  and  one  year  (1906-07)  at  the 
University  of  Illinois,  in  Urbana.  At  the  University  of  Illinois, 

Dodd  had  become  acquainted  with  Rietz,  who  at  that  time  was  dividing 
his  time  about  equally  between  his  position  of  Assistant  Professor 
of  Mathematics  in  the  Department  of  Mathematics,  and  his  position 

of  Statistician  in  the  Experiment  Station  of  the  College  of 
Agriculture.  Rietz  was  teaching  a course  in  the  Mathematics  Depart- 
ment entitled  "Averages  and  Mathematics  of  Investment",  which  he  had 
been  induced  to  develop  two  years  before,  when  a demand  had  arisen 
for  a course  in  statistics  which  none  of  the  members  of  the  Mathe- 
matics Department  were  particularly  prepared  to  give.  Also,  at  that 
time  Rietz  was  very  busy  working  on  his  first  publication  in 
statistics,  a 32  page  appendix  ("Statistical  Methods.  Appendix  to 
Principles  of  Breeding")  to  A Treatise  on  Thremmatology  by  Eugene 
Davenport,  Dean  of  the  College  of  Agriculture  and  Director  of  the 
Agricultural  Experiment  Station  (Boston:  Ginn  and  Co.,  1907, 
pp.  681-713);  and  also  on  his  bulletin  (with  Dean  Davenport)  on 
Statistical  Methods  Applied  to  the  Study  of  Type  and  Variability 
in  Com  (Illinois  Agriculture  Experiment  Station  Bulletin  No.  119, 
1907).  From  then  until  he  was  called  to  the  University  of  Iowa  in 
1918  as  Head  of  the  Department  of  Mathematics,  Rietz  published  a 
long  list  of  papers  on  statistical  topics,  some  purely  theoretical, 
some  expositional,  some  arising  out  of  his  connection  with  the 
College  of  Agriculture.  I mention  these  details  to  emphasize  the 
fact  that  the  development  of  statistical  theory  and  methodology 
in  the  United  States  owes  far  more  to  the  needs  and  support  of 
workers  In  agriculture  than  many  people  realize  today . 
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5.  Under  Rietz's  leadership  the  University  of  Iowa  rapidly  became  one 
of  the  leading  centers  of  actuarial  mathematics  in  the  United  States, 
and  the  leading  center  for  research  In  mathematical  statistics. 

(Other  notable  centers  of  actuarial  mathematics  and  mathematical 
statistics  were  the  University  of  Michigan,  in  Ann  Arbor,  under  the 
leadership  of  James  W.  Glover  (1868-1941)  and  Harry  C.  Carver,  who 

in  1930  founded,  and  for  five  years  personally  financed  the  Annals 
of  Mathematical  Statistics;  and  Harvard  University,  under  the  leader- 
ship  of  Edward  V,  Huntington  (1874-1952),  Truman  L.  Kelley  (1884-1961) 
and  Warren  M.  Persons  (1878-1937).)  Two  of  Rietz's  publications  helped 
to  firm  up  the  University  of  Iowafs  standing:  (1)  the  Handbook  of 
Mathematical  Statistics  (Boston:  Houghton  Mifflin  Company,  1924) 
prepared  by  the  "Members  of  the  Committee  on  the  Mathematical  Analysis 
of  Statistics  of  the  Division  of  Physical  Sciences  of  the  National 
Research  Council"  (H.  C.  Carver,  A.  R.  Crathome,  W.  L.  Crum,  James 
W.  Glover,  E.  V.  Huntington,  Truman  L.  Kelley,  Warren  M.  Persons, 

H.  L.  Rietz,  and  Allyn  A.  Young)  with  Rietz  serving  as  Editor-in-Chief; 
and  (2)  Rietz's  own  Carus  Mathematical  Monograph  (No.  3)  entitled 
Mathematical  Statistics,  published  for  the  Mathematical  Association 
of  America  by  the  Open  Court  Publishing  Company  in  1927,  which  served 
as  the  basis  for  courses  in  mathematical  statistics  given  in  Depart- 
ments of  Mathematics  of  many  universities  and  colleges  for  years 
afterward.  The  jointly  written  Handbook  was  doomed,  however,  to 
become  obsolete  almost  upon  publication:  the  future  of  mathematical 

statistics  was  being  shaped  in  the  1920's  by  the  papers  of  R.  A. 

Fisher;  and  the  future  of  statistical  methodology,  by  his  Statistical 
Methods  for  Research  Workers  (1925) , which  rapidly  became  "the  Bible" 
of  statistical  methodology  Iowa  State  College,  Ames,  under  the 
guidance  of  Professors  George  W.  Snedecor  (1881-1974)  and  A.  E. 

Brandt.  (For  additional  information  on  Rietz,  see  A.  R.  Crathome, 

"Henry  Lewis  Rietz — In  Memoriam",  Annals  of  Mathematical  Statistics, 

Vol.  15,  No.  1 (March  1944),  102-108,  which  contains  lists  of 
selected  publications  of  Rietz,  of  his  books,  and  of  doctorate 
dissertations  written  under  his  supervision;  and  Frank  Mark  Weida, 

"Henry  Lewis  Rietz,  1875-1943",  Journal  of  the  American  Statistical 
Association,  Vol.  39,  No.  226  (June  1944),  249-250.) 

6.  After  one  year  of  graduate  work  in  actuarial  mathematics  at  Iowa, 

Curtiss  decided  against  a career  as  an  actuary,  and  went  on  to  earn 
his  Ph.D.  in  pure  mathematics  (analysis)  at  Harvard  in  1935.  However, 
five  years  later,  as  instructor  in  mathematics  at  Cornell  University, 
and  the  most  junior  member  of  the  Mathematics  Department,  he  was 
assigned  the  responsibility  of  a course  in  mathematical  statistics. 

To  prepare  for  this  course,  to  answer  the  teasing  query  of  his  senior 
colleagues,  "What  is  there  to  statistics  anyway?",  he  dug  into  the 
first  ten  volumes  of  the  Annals  of  Mathematical  Statistics,  the 
first  six  volumes  of  the  Supplement  to  the  Journal  of  the  Royal 
Statistical  Society  (borrowed  from  the  late  Frederick  F.  Stephan 
(1903-1971),  and  J.  0.  Irwin's  series  of  reviews  of  "Recent  Advances 
in  Mathematical  Statistics"  in  the  Journal  of  the  Royal  Statistical  Society, 
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and  other  sources.  In  the  third  of  these  reviews  (for  1932),  he 
no  doubt  noticed  nine  of  the  fourteen  pages  of  the  section  on 
"Exact  sampling  distributions"  were  devoted  to  discussion  of  four 
papers  of  his  friend  Sam  Wilks.  During  World  War  II,  Curtiss,  as 
a Lt.  Commander,  USNR,  applied  modern  statistical  theory  and 
methodology  to  problems  of  naval  engineering  with  considerable 
success  in  the  Bureau  of  Ships  of  the  U.S.  Navy  Department.  (For 
discussion  of  some  of  these  applications,  see  J.  H.  Curtiss 
"Statistical  Inference  Applied  to  Naval  Engineering",  Journal  of 
the  American  Society  of  Naval  Engineers,  Vol.  58,  No.  3 (August 
1946),  335-398.)  In  April  1946,  he  was  brought  to  the  National 
Bureau  of  Standards  by  its  new  Director,  Dr.  E.  U.  Condon  (1902-1974), 
and  appointed  statistical  assistant  to  the  Director  for  the  express 
purpose  of  introducing  modern  statistical  theory  and  methodology 
into  the  scientific  and  technical  programs  of  the  Bureau.  However, 
before  Curtiss  could  get  such  a program  under  way.  Dr.  Condon  was 
obliged  to  turn  over  to  him  the  day-to-day  administration  of  the 
Bureau's  new  responsibilities  in  the  development  of  large-scale 
automatic  digital  computers,  and  of  an  associated  program  of  developing 
the  mathematics  of  numerical  analysis.  John's  original  assignment 
at  the  Bureau  was  therefore  placed  on  my  shoulders,  when  I arrived 
at  the  Bureau  to  receive  it  on  October  1,  1946 — and  the  rest  of  that 
story  you  know. 

7.  See  WILKS  1941,  1942,  1948;  pp.  18-19  of  ANDERSON  1965;  and 
items  (40),  (41),  and  (45)  in  the  list  of  "The  Publications  of 
S.  S.  Wilks"  appended  thereto. 

8.  Rietz  gave  a paper,  "Comments  on  Applications  of  Recently  Developed 
Theory  of  Small  Samples",  at  the  92nd  Annual  Meeting  of  the  American 
Statistical  Association,  Cleveland,  Ohio,  30  December  1930,  which  saw 
publication  in  the  Journal  of  the  American  Statistical  Association, 

Vol.  26,  No.  175  (June  1931),  150-158. 

9.  Thus  Paul  Rider,  in  a valuable  review  article,  A Survey  of  the 
Theory  of  Small  Samples  (Annals  of  Mathematics,  2nd  Series , 

Vol.  31,  No.  4,  (October  1930),  pp.  577-628),  which  was  later  to 
"save  my  neck"  on  a number  of  occasions,  wrote  (p.  578): 

"Undoubtedly  the  leading  writer  in  the  theory 
of  small  samples  is  R.  A.  Fisher,  whose  work  in 
this  field  has  revolutionized  modern  sampling  theory. 

Much  of  it  is  to  be  found  In  his  book,  Statistical 
Methods  for  Research  Workers , but  this  book  is 
extremely  unsatisfying  to  a mathematician,  as  it 
merely  states  results  without  proofs  and  usually 
without  even  indicating  how  a given  result  may  be 
derived.  It  discusses  such  things  as  the  distribution 
of  t without  telling  what  the  distribution  is.  His 
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original  papers  are  much  more  enlightening,  but 
from  the  references  as  given  in  the  book  it  is 
sometimes  difficult  to  tell  which  paper  treats 
of  a given  topic.  Even  these  papers  suffer  in 
places  from  the  same  defects  as  those  of  the  book, 
and  they  are  often  troublesome  to  follow." 

I don't  know  whether  Paul  later  retracted  these  remarks,  or 
Fisher  was  forgiving,  because,  when  I got  to  University  College, 
London,  in  1935,  to  study  under  J.  Neyman  and  E.  S.  Pearson, 
there  was  Paul  sitting  at  a desk  up  in  "Fisher  territory"  (the 
Galton  Laboratory  and  Department  of  Eugenics) , working  on  moment 
functions  for  Fisher's  k-statistlcs  in  samples  from  a finite 
population. 

10.  Hotelling's  paper  on  "The  distribution  of  correlation  ratios 
calculated  from  random  data",  in  Proceedings  of  the  National 
Academy  of  Sciences , 11,  no.  10  (October  1925),  657-662,  made 
him  the  first  person  in  the  United  States  to  respond  in  kind  to 
R.  A.  Fisher's  signal  contributions  to  the  theory  of  small  samples — 
his  derivation  employed  the  same  kind  of  geometrical  reasoning  in 
terms  of  Euclidean  N-dimensional  space  that  Fisher  had  used  so 
effectively.  This  paper  carries  a footnote  that  I've  always 
considered  to  be  very  significant.  I believe  it  affords  an  explana- 
tion of  why  so  many  American  mathematicians  had  difficulty  following 
Fisher's  geometrical  proofs.  Anyone  who  attempts  to  duplicate 
Fisher's  geometrical  reasoning  soon  discovers  that  a crucial  step 
is  the  correct  evaluation  of  the  relevant  element  of  volume. 
Hotelling,  at  this  juncture  in  his  paper,  gives  a general  expression 
for  the  relevant  element  of  volume,  which  he  numbers  "(17)",  and 
then  remarks  in  a footnote; 

"This  important  expression  for  the  volume  element  has  been  used  in 
lectures  by  [at  Princeton  University]  by  Professors  0.  Veblen  and 

L.  P.  Eisenhart.  I do  not  find  it  in  any  of  the  treatises  on 
Calculus,  Analysis  or  Differential  Geometry,  save  for  the  special 
case  in  which  the  manifold  of  integration  is  a surface.  It  may 
readily  be  proved  by  showing  first  that  (17)  is  a relative  invariant 
under  arbitrary  transformations  of  the  parameters;  and  second,  that 
if  the  parameters  of  the  hypersurface  are  orthogonal  at  a point, 

(17)  becomes  at  this  point  the  simple  expression  for  the  volume 
element  in  cartesian  coordinates." 

Hotelling  had  gone  to  Princeton  University  as  a J.S.K.  Fellow  in 
mathematics,  1921-1922,  after  receiving  his  A.  B.  (1919)  and  an 

M. S,  (1921)  from  the  University  of  Washington,  in  Seattle.  His 
interests  in  statistics  predated  his  going  to  Princeton  in  the  Fall 
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of  1921.  He  had  hoped  to  find  some  work  in  probability  theory  and 
the  mathematics  of  statistics  going  on  there  in  the  Mathematics 
Department.  Finding  none,  he  undertook  instead  a program  of  study 
and  research  in  topology  (then  called  "analysis  situs")  and 
differential  geometry,  under  the  direction  of  Professor  Oswald 
Veblen  (1880-1960)  and  my  father,  Luther  Pfahler  Elsenhart  (1876-1965) . 
He  stayed  on  at  Princeton,  1922-1924,  as  an  Instructor  in  Mathematics 
and  received  his  Ph.D.  from  Princeton  University  in  June  1924,  his 
doctoral  dissertation  being  on  "Three-dimensional  manifolds  of 
states  in  motion."  In  1927  he  published  a paper  "An  application  of 
analysis  situs  to  statistics"  (Bulletin  of  the  American  Mathematical 
Society,  Vol.  33,  (1927),  pp.  467-476),  which  had  to  do  with 
topological  aspects  of  serial  and  multiple  correlations. 

Following  receipt  of  his  Ph.D. , Hotelling  returned  to  the  West 
Coast,  to  Stanford  University,  where  he  was  a Junior  Research 
Associate  (1924-25) , and  then  Research  Associate  (1925-27) , in  the 
Food  Research  Institute;  and  finally,  an  Associate  Professor  of 
Mathematics  (1927-31),  in  the  Department  of  Mathematics.  Hotelling 
visited  Fisher  in  England,  in  1929,  hoping  to  persuade  Fisher  to 
join  with  him  in  the  preparation  of  an  up-to-date  textbook  on  the 
mathematics  of  Statistical  Inference.  Fisher  was  not  Interested  in 
the  proposition.  In  1931,  Hotelling  was  called  to  Columbia  University, 
in  New  York  City,  as  Professor  of  Economics  to  develop  further  the 
existing  work  there  in  Mathematical  Economics,  and  to  initiate  a 
program  in  Mathematical  Statistics. 

11.  These  papers  had  been  followed  by  their  more  elegantly  written  "On 
the  problem  of  two  samples"  (Bulletin  de  l’Academie  Polonaise  et 
des  Lettres,  Series  A,  1930,  471-494),  and  "On  the  problem  of  k 
samples"  (idem,  1931,  460-481),  in  which  the  likelihood-ratio 
technique  had  led  directly  to  the  now  famous  test  for  the  homogeneity 
of  variance  involving  the  ratio  of  the  weighted  arithmetic  mean  of 
the  sample  variances  (with  weights  subsequently  modified  by  Bartlett) . 
This  great  discovery  was  discussed  by  Pearson  in  one  of  his 
lectures,  and  no  doubt  contributed  to  Sam’s  enthusiasm  for  likelihood- 
ratio  tests. 

It  was  too  early  to  claim  that  the  tests  thus  found  were  "best"  in 
some  sense  Inasmuch  as  the  Neyman-Pearson  Lemma  was  yet  to  come 
in  J,  Neyman  and  E.  S.  Pearson,  "On  the  problem  of  the  most  efficient 
tests  of  statistical  hypothesis", communicated  to  the  Royal  Society 
of  London  in  August  1932,  "read"  to  the  Society  on  November  10,  1932, 
and  published  on  February  16,  1933  in  the  Society's  Philosophical 
Transactions , Series  A,  Vol.  231,  pp.  289-337;  which,  incidentally 
was  refereed  by  Fisher  who,  at  the  time,  considered  it  an  important 
step  forward. 
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12.  Time  and  again  during  his  years  at  the  Bureau  I would  hear  him  tell 

a consultee,  or  an  audience,  that  he  was  "a  chemist'1,  implying  that 
he  was  not  a statistician.  Well,  Jack  may  have  been  all  chemist  at 
one  time,  but  by  1931  he  was  already  on  his  way  to  becoming  an 
exponent  and  practitioner  of  Fisherian  methods  too.  He  had  come 
upon  Student’s  t test  "by  accident .. .in  1925"  (W.  J,  Youden,  Risk, 

Choice  and  Prediction:  An  Introduction  to  Experimentation,  Duxbury  Press, 
North  Scituate,  Mass.,  1974,  p.  5).  By  the  "summer  of  1931  [he]  had 
obtained  one  of  the  1050  copies  printed  of  the  first  edition"  of 
Fisher's  Statistical  Methods  for  Research  Workers  (1925),  and  when 
Fisher  "visited  Cornell"  to  attend  the  6th  International  Congress  of 
Genetics,  24-31  August  1931,  Youden  "drove  there... to  show  him  an 
experimental  arrangement".  (Quotations  are  from  p.  727  of  W.  J. 

Youden,  "Memorial  to  Sir  Ronald  Aylmer  Fisher,"  Journal  of  the  American 
Statistical  Association,  Vol.  57,  No.  300  (Dec.  1962),  727-728.) 

From  Hotelling’s  lectures  Youden  "first  got  some  hint  that  [Fisher's 
Statistical  Methods  ...  3 also  held  a message  for  mathematicians .. .He 
told  the  young  men  listening  to  him  not  to  be  misled  by  the  large 
print,  the  wide  margins,  and  a text  almost  devoid  of  mathematical 
symbols,  that  in  this  book  were  concepts  as  new  to  the  theorists  as 
to  the  researchers".  (Quoted  from  p.  47  of  W.  J.  Youden,  "The  Fisherian 
Revolution  in  Methods  of  Experimentation,"  Journal  of  the  American 
Statistical  Association,  Vol.  46,  No.  253  (March  1951),  47-50.) 

During  the  next  few  years  he  published  a variety  of  papers  expounding 
and  demonstrating  the  application  of  known  statistical  techniques  to 
various  problems  arising  in  studies  of  apples,  seeds,  soils,  leaves, 
tomatoes,  trees  and  viruses.  He  had  clearly  "crossed  the  Rubicon"; 
was  on  his  way  to  becoming  an  expert  expositor  and  practitioner  of 
statistical  methods  in  experimentation;  and  from  then  on  he  became 
more  and  more  of  a statistician — or  shall  we  say,  "experimentrician" — 
and  less  and  less  "chemist". 

13.  Spearman  devoted  over  40  years  of  his  life  to  the  development  of  a 
psychological  theory  of  mental  ability  built  around  a General  Factor, 
g,  that  characterizes  an  individual's  "general  mind  power" — see 

his  The  Abilities  of  Man  (New  York:  The  Macmillan  Company,  1972); 

but  is  most  widely  known  among  statisticians  today  for  a comparatively 
minor  contribution,  his  coefficient  of  rank-order  correlation  (1904) . 

14.  Whether  the  population  tetrad  differences,  TJ234  **  P.12P34  * P13P24 

t 1324  * P13P24  “ Pi 4P23 » were  both  zero,  both  non-zero,  or  one  zero  and  the 
other  non-zero,  where  p. . is  the  coefficient  of  correlation  between 
the  i-th  and  j-th  traits,  was  of  decisive  importance  in  Spearman's 
theory  of  mental  abilities  of  man. 

15.  Rietz  had  chaired  the  session  on  Statistical  Methodology  on  the  first: 
day  of  the  92nd  Annual  Meeting  of  the  American  Statistical  Association 
in  Cleveland,  Ohio,  December  29-31,  1930,  at  which  Shewhart  had 
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presented  his  paper  on  ’’Statistical  Method  from  an  Engineering 
Viewpoint”  (published  in  the  Proceedings  of  the  Meeting  as  "Applica- 
tions of  Statistical  Method  in  Engineering",  Journal  of  the  American 
Statistical  Association,  Vol.  26,  March  1931  Supplement,  pp.  214-221); 
and  the  following  day  Shewhart  had  been  the  invited  discussant  of 
Hotelling’s  paper  on  "Recent  Improvements  in  Statistical  Inference" 
(same  Supplement,  pp.  79-87;  discussion,  pp.  87-89). 

16.  This  was  Karl  Pearson’s  last  year  as  the  first  Galton  Professor 

of  National  Eugenics  (1911-1933) , as  Editor  of  the  Annals  of  Eugenics, 
which  he  had  founded  and  edited  since  1925,  and  as  Head  of  the 
Department  of  Applied  Statistics  (1911-1933)  , which  included  the 
Biometric  Laboratory  (which  Pearson  had  originated  in  1895,  as  a 
center  for  postgraduate  study  in  this  new  branch  of  applied  mathe- 
matics when  Goldsmid  Professor  of  Applied  Mathematics  and  Mechanics 
(1884-1911))  and  the  Francis  Galton  Laboratory  of  National  Eugenics 
(which  had  been  formed,  and  placed  under  Pearson's  direction,  in 
1906  at  Galton’ s request,  as  successor  to  Galton ’s  own  Eugenics 
Records  Office  established  at  University  College  in  1904  by  a gift 
from  Galton  to  the  University  of  London  for  this  purpose) . He 
continued,  however,  to  edit  Biometrika,  of  which  he  was  one  of  the 
three  founders,  always  the  principal  editor  (vols.  1-28,  1901-1936), 
and  for  many  years  the  sole  editor;  and  had  almost  seen  the  final 
proofs  of  the  first  half  of  volume  28  through  the  press  when  he 
died  on  27  April  1936. 

When  I arrived  at  University  College  in  October  1935  as  a Ph.D. 
candidate  in  statistics,  we  were  told  that  Karl  Pearson's  strength 
was  rapidly  failing,  that  he  was  still  driving  himself  to  shut  out 
his  grief  over  the  thwarting  of  his  ideal  of  an  Applied  Statistics 
Institute  (with  Readers  in  Genetics,  Medicine,  Psychology,  Mathematical 
Statistics,  etc.)  by  the  break  up  of  his  Department  into  separate 
Departments  of  Eugenics  and  Applied  Statistics;  and  that  he  was  very 
reluctant  to  see  visitors.  The  end  came  before  Paul  Rider  and  I 
and  many  of  our  fellow  students  were  granted  opportunities  to  meet 
him.  I have  never  quite  recovered  from  that  lost  opportunity, 

17,  E.  S.  Pearson  had  spent  some  time  with  Shewhart  and  his  colleagues 
at  the  Bell  Telephone  Laboratories  during  his  1931  visit  to  the 
United  States . He  was  one  of  the  early  exponents  in  England  of 
Shewhart 's  control-chart  techniques,  and  at  the  time  of  Sam's  visit 
was  engaged  in  the  preparation  of  a paper  on  "Statistical  Method 

in  the  Control  and  Standardization  of  the  Quality  of  Manufactured 
Products",  presented  at  the  December  1932  meeting  of  the  Royal 
Statistical  Society,  and  later  published  in  the  Society’s  Journal , 

(Vol.  96  (1933),  pp.  21-60).  This  paper  was  largely  responsible 
for  the  formation  of  the  Industrial  and  Agricultural  Research 
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Section  of  the  Royal  Statistical  Society  on  November  23,  1933,  and 
the  subsequent  publication  of  the  now-famous  Supplement  to  the 
Journal  of  the  Royal  Statistical  Society  to  provide  a medium  for 
publication  of  papers  of  this  Section.  (For  further  details, 
see  E.  S.  Pearson,  "Some  Historical  Reflections  on  the  Introduction 
of  Statistical  Methods  in  Industry:  The  Statistician,  Vol.  22,  No.  3 

(Sept.  1973),  165-179.) 


18.  Stanley,  like  his  father,  received  an  A.B, — but  in  mathematics, 
not  architecture — from  North  Texas  State  College  ("Teacher fs" 
having  been  dropped  from  the  name)  in  1955.  He  studied  at 
Cambridge  University  1955-1956;  married  Jocelyn  Wilkins,  daughter 
of  a classmate  of  Sam's  at  North  Texas  State,  in  1958;  received  an 
M.S.  in  applied  mathematics  from  Columbia  University  in  1961; 

has  three  daughters  and  a son;  and  works  for  the  Department  of 
Defense  as  a mathematician. 

19.  John  Wishart  had  gained  First  Class  Honors  Degree  in  Mathematics 
and  Natural  Philosophy  at  the  University  of  Edinburgh,  in  Scotland, 
in  1922.  At  Edinburgh  he  had  attended  the  lectures  of  E.  T. 

Whittaker  (1873-1956),  on  "The  Calculus  of  Observations"  which 
were  later  to  appear  in  book  form  (T.  WHITTAKER  and  ROBINSON, 

The  Calculus  of  Observations,  London  and  Glasgow:  Blackie  and 

Son,  Ltd.,  1924),  and  had  learned  numerical  mathematics  "the  hard  way", 

i.e.,  without  the  benefit  of  a desk  calculator,  in  Whittaker’s  Mathematical 

Laboratory.  In  the  autumn  of  1924,  Wishart  had  joined  Karl 

Pearson  at  University  College,  as  a Research  Assistant.  One  of 

Wishart Ts  main  tasks  on  arriving  there  was  to  get  work  on  Pearson’s 

Tables  of  the  Incomplete  Beta-Function  underway. 

Wishart  stayed  with  Pearson  for  three  years  and  then  in  the  autumn 
of  1927  accepted  a teaching  position  at  the  Imperial  College  of 
Science  and  Technology  (of  the  University  of  London),  inasmuch  as 
he  was  a teacher  by  training  and  temperament.  While  still  with 
Pearson  he  had  collaborated  with  R.  A.  Fisher  on  a joint  paper 
"On  the  distribution  of  the  error  of  an  interpolated  value  and  on 
the  construction  of  tables"  (Proceedings  of  the  Cambridge  Philosophical 
Society,  Vol.  23,  Part  8 (October  1927),  pp.  917-921).  He  was 
barely  settled  in  his  new  post  at  Imperial  College,  when,  at  the 
beginning  of  1928,  he  was  offered  and  accepted  an  appointment  as 
Statistical  Assistant  to  R.  A.  Fisher  at  Rothamsted  Experiment 
Station.  With  Fisher's  encouragement,  he  derived 
"The  generalized  product  moment  distribution  in  samples  from 
a normal  multivariate  population"  ( Biometrika,  Vol.  22A,  Parts  1&2 
(July  1928),  pp . 31-52),  by  a geometrical  argument  analogous  to 
those  used  previously  by  Fisher,  the  simultaneous  distribution  of 
the  sample  estimates  of  the  variances  and  covariances  of  a multi- 
variate normal  population  corresponding  to  a sample  of  N items  from 
such  a population,  and  prepared  an  extensive  tabulation  of  the 
moments  and  product  moments  of  this  distribution,  which  is  now  known 
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as  "Wishart 's  distribution".  Wishart,  during  his  three  years  at 
Rothamsted  (1928-31)  participated  fully  not  only  in  the  mathematical 
research  on  sampling  distributions  and  their  properties,  but  also 
in  the  advisory  and  service  activities  of  Rothamsted  Statistical 
Department  during  that  period,  as  reflected  by  the  twenty  publications 
of  which  he  was  the  single  or  joint  author  during  this  period. 

In  October  1931,  a few  months  after  G.  U.  Yule's  retirement  from 
full-time  teaching  as  Reader  in  Statistics  in  the  University  of 
Cambridge,  Wishart  was  appointed  to  a newly  created  post  of  Reader 
in  Statistics  in  the  Faculty  of  Agriculture,  with  responsibilities 
also  for  some  teaching  in  the  Faculty  of  Mathematics.  This  was  an 
exceptionally  fine  appointment:  at  Cambridge,  as  at  other  English 

universities,  a Readership  is  only  one  step  below  a Professorship, 
and  until  the  late  1950 's  Professorships  were  very  few  and  far 
between,  there  ordinarily  being  only  one  per  established  discipline 
(e.g..  Mathematics),  which  Statistics  certainly  was  not  at  that 
time.  (Thus  Yule  himself  had  been  merely  a University  lecturer  in 
Statistics  from  1912  until  only  a few  months  before  his  premature 
retirement  owing  to  ill  health) . Wishart  saw  in  his  Cambridge 
appointment  an  opportunity  to  introduce  statistics  to  mathematical 
undergraduates,  and  began  at  once  to  offer  not  only  a general 
course  on  statistical  methods  in  the  Faculty  of  Agriculture,  but 
also  a course  on  mathematical  statistics  which  undergraduate  students 
in  the  Faculty  of  Mathematics  could  offer  for  Schedule  B of  the 
Mathematical  Tripos.  Among  his  early  students  in  this  program  were 
M.  S.  Bartlett  (B.  A.,  Queens'  College,  1932)  and  W.  G,  Cochran 
(B.A.,  St.  John's  College,  1933).  (For  additional  information  on 
Wishart,  see  E.  S.  Pearson,  "John  Wishart,  1898-1956",  Biometrika, 

Vol.  44,  Pts.  1&2  (June  1957),  1-8,  which  includes  a bibliography  of 
his  published  work;  and  M.  S.  Bartlett,  "John  Wishart,  D.Sc., 

F.R.S.E.",  Journal  of  the  Royal  Statistical  Association,  Series  A, 

Vol.  119,  Pt.  4 (1956),  492-493.) 

20.  A few  words  are  in  order  on  how  my  father  became  interested  in, 
and  partial  to  statistics. 

My  father's  primary  mathematical  interest  was  differential  geometry, 
and  his  research  was  exclusively  in  that  area.  Exactly  when  he  began 
to  take  an  "outside"  interest  in  mathematical  statistics  I do  not 
know.  It  may  have  been  as  early  as  1913,  when  as  noted  earlier,  he 
corresponded  with  Edward  L,  Dodd  on  various  aspects  of  the  latter's 
paper  entitled  "The  probability  of  the  arithmetic  mean  compared  with 
that  of  certain  other  functions  of  measurements",  which  was  published 
in  the  Annals  of  Mathematics  (Vol.  14,  pp.  186-198,  June  1913),  of  which 
my  father  was  then  an  editor.  At  any  rate,  thereafter  Dodd  sent  my 
father  reprints  of  many  of  his  subsequent  papers  on  functional  and 
statistical  properties  of  various  types  of  "means",  which  my  father  kept 
and  ultimately  turned  over  to  me  when  I became  interested  in  such 
matters  in  the  early  '30's. 
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Early  in  1924,  "at  the  request  of  the  Commission  on  New  Types  of 
Examination  of  the  College  Entrance  Examination  Board",  my  father 
"formed  a committee  of  mathematicians  to  examine  critically  certain 
statistical  methods  used  in  the  investigations  of  the  Commission" 
(American  Mathematical  Monthly,  Vol.  31,  No.  4 (April  1924),  p.  209). 
The  '^mathematicians " of  the  Committee  included  the  economic 
statisticians  W.  Randolph  Burgess  and  W.  L.  Crum  (1894-1967)  of  the 
Federal  Reserve  System  and  Economics  Department,  Harvard,  respectively; 
the  mathematicians  E.  V.  Huntington  (1874-1952)  and  J.  H.  M. 

Wedderburn  (1882-1948),  of  Harvard  and  Princeton,  respectively;  and 
the  mathematical  statistician,  H.  L.  Rietz. 

The  findings  of  this  Committee,  my  father’s  continued  advisory 
relations  with  the  higher-ups  of  the  College  Entrance  Examination 
Board  (CEEB),  and  Wilks's  contributions  at  Iowa  (and  under  Hotelling 
at  Columbia)  to  the  solution  of  statistical  problems  arising  in 
educational  testing,  made  it  possible  for  my  father  to  arrange  a 
part-time  appointment  with  the  CEEB  concurrent  with  his  initial 
University  appointment — a relationship  with  the  Board,  and  its 
successor,  the  Educational  Testing  Service,  that  continued  until 
Wilks’s  death. 

As  mentioned  earlier.  Hotelling,  after  receiving  his  Ph.D.  in 
mathematics  from  Princeton  in  1924,  went  to  Stanford  University, 
first  to  a position  in  the  Stanford  Food  Research  Institute,  later 
in  the  Mathematics  Department,  Stanford  University,  During  these 
years  at  Stanford  (1924-1931)  he  wrote  and  published  a stream  of 
important  original  contributions  to  statistical  theory  and  mathe- 
matical economics;  reviews  of  American  and  English  books  on 
statistical  methods,  (e.g.,  of  Statistical  Analysis  by  Edmund 
E.  Day  (New  York:  The  Macmillan  Company,  1925),  in  Journal  of  the 
American  Statistical  Association,  Vol.  21,  No.  155  (Sept.  1926), 
360-363),  In  which  he  deplored  the  obsoleteness  of  teaching  and 
research  in  statistics  in  the  United  States  and  placed  the  blame 
squarely  on  the  doorsteps  of  Departments  of  Mathematics;  and 
expository  articles  on  "British  statistics  and  statisticians  today" 
(Journal  of  the  American  Statistical  Association,  Vol,  25,  No.  170 
(June  1930),  186-190),  "Recent  improvements  in  statistical 
inference"  (cited  fully  in  footnote  15),  etc.,  in  which  he  did  his 
very  best  to  acquaint  American  readers  with  the  "new  look"  in 
statistics.  He  regularly  sent  reprints  of  all  of  these  to  my 
father.  When  my  father  gave  them  to  me  in  the  Fall  of  1932,  as 
I was  reading  up  on  "Student-Fisher  statistics",  it  was  quite  clear 
that  my  father  had  more  than  a superficial  knowledge  of  the  papers 
on  statistical  theory,  and  had  "got  the  message"  of  Hotelling's 
book  reviews  and  expository  articles. 
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21.  This  assignment  was  very  disruptive  to  Duncan  at  that  time.  When 
asked  to  undertake  it  he  was  already  at  work  on  his  doctoral 
dissertation  on  "South  African  gold  and  international  trade";  and 
his  acceptance  of  it  delayed  until  1936  his  completion  of  the 
requirements  for  his  Ph.D.  in  Economics.  He  also  lost  out  on  one 
of  the  features  that  "sweetened"  the  proposition,  an  opportunity 
to  visit  the  West  Coast — when  the  plans  were  made.  Hotelling  was 
at  Stanford  University,  but  had  moved  on  to  Columbia  University 
before  the  time  arrived  for  Duncan  to  study  under  him.  This 
assignment  was  to  be  instrumental  in  changing  the  direction  of 
Duncan’s  subsequent  career. 

22.  The  aim  of  the  Department  of  Economics  and  Social  Institutions 
was  to  improve  its  own  offerings  in  statistics  for  economics 
students  by  integrating  and  updating  the  Smith-Duncan  sequence 
of  courses  within  that  department.  The  extent  to  which  this  aim 
was  achieved  is  evidenced  by  the  two  volumes  Fundamentals  of  the 
Theory  of  Statistics;  Vol.  1,  Elementary  Statistics  and  Applications; 
Vol.  2,  Sampling  Statistics  and  Applications,  authored  jointly  by 
Professors  Smith  and  Duncan  and  published  by  the  McGraw-Hill  Book 
Company,  Inc.,  in  1944,  1945,  respectively.) 

23.  For  further  details  on  the  founding  and  early  years  of  the 
Annals  of  Mathematical  Statistics  see  the  letter  from  Harry  C. 

Carver,  dated  14  April  1972,  to  Professor  [W.  J.j  Hall,  reproduced 
in  the  Institute  of  Mathematical  Statistics  Bulletin,  2,  No.  1 
(Jan.  1973),  11-14;  and  Allen  T.  Craig,  "Our  Silver  Anniversary", 
in  Annals  of  Mathematical  Statistics,  31,  no.  4 (Dec.  1960), 

835-837. 

24.  The  material  of  the  five  following  paragraphs  is  taken  for  the 

most  part  from  MALONEY  1962  and  SIMON  1965,  where  further  details  can 
be  found  on  the  history  of  statistical  methodology  in  Army  research, 
development  and  testing. 

25.  See  Proceedings  of  the  First  Symposium  on  Statistical  Methods: 

Sampling  Techniques  (4-5  November  1953),  Ballistic  Research 
Laboratories  Report  No.  897,  Aberdeen  Proving  Ground,  Maryland, 

January  1954. 

26 * Proceedings  of  the  First  Conference  on  the  Design  of  Experiments 
in  Army  Research,  Development  and  Testing,  Office  of  Ordnance 
Research  Report  No.  57-1,  Office  of  Ordnance  Research,  Durham, 

North  Carolina,  June  1957. 
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W.  G.  COCHRAN*  "The  Philosophy  Underlying  the  Design  of 
Experiments." 

CHURCHILL  EISENHART,  "The  Principle  of  Randomization  in  the 
Design  of  Experiments," 

M.  E.  TERRY,  "Finding  Optimum  Conditions  by  Experimentation." 

W.  J.  YOUDEN,  "Design  of  Experiments  in  Industrial  Research  and 
Development. " 

Panel  Discussion  on  "How  and  Where  Do  Statisticians  Fit  In." 

The  Panel:  John  W.  Tukey,  Chairman,  Cuthbert  Daniel,  Besse  Day, 

Churchill  Eisenhart,  M.  E.  Terry,  and  S.  S.  Wilks. 
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R.  A.  FISHER,  "Practical  Problems  in  Experimental  Design." 
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A.  W,  MARSHALL,  "Experimentation  by  Simulation  and  Monte  Carlo." 
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A.  W,  KIMBALL,  "Errors  of  the  Third  Kind  in  Statistical  Consulting. 
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G.  A.  WATTERSON,  ’'Time  Series  and  Spectral  Analysis.’1 

J.  M.  HAMMERSLEY,  ’’Monte  Carlo  Methods.” 

R.  L.  ANDERSON,  "Designs  for  Estimating  Variance  Components.” 
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8th : Washington,  D.C.,  24-26  October  1962. 

Walter  Reed  Army  Institute  of  Research,  Walter  Reed  Army  Medical  Center. 

EGON  S.  PEARSON,  "A  Statistician's  Place  in  Assessing  the  Likely 
Operational  Performance  of  Army  Weapons  and  Equipment." 

MARVIN  A.  SCHNEIDERMAN,  "A  General  Survey  of  Screening  Theory." 

HERMAN  CHERNOFF,  "Optimal  Design  Experiments.” 

R.  P.  ABELSON,  "An  Experimental  Design  for  Decisions  Under 
Uncertainty. " 

H.  C.  BATSON,  "Bio-Assay , " 
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Directorate  of  Research  and  Development,  U.S.  Army  Missile  Command. 

SOLOMON  KULLBACK,  "Communication  Theory." 

FRANK  PROS CHAN,  "The  Concept  of  Monotone  Hazard  Rate  in  Systems 
Reliability." 

CHURCHILL  EISENHART,  "Realistic  Evaluation  of  the  Precision  and 
Accuracy  of  Instrument  Calibration  Systems." 
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Army  Research  Office,  Office  Chief  of  Research  and  Development,  Departmen 
of  the  Army. 

MAJ.  GEN.  LESLIE  E.  SIMON  (Rat’d),  "The  Stimulus  of  S.  S.  Wilks  to 
Army  Statistics." 


OSCAR  KEMPTHORNE,  "Development  of  the  Design  of  Experiments  Over 
the  Past  Ten  Years." 

H.  0.  HARTLEY  and  A.  W.  WORTHAM,  "Assessment  and  Correction  of 
Deficiences  in  Pert  Analysis." 

CHURCHILL  EISENHART,  "Sam  Wilks  as  I Remember  Him." 

W.  J.  YOUDEN,  "An  Operations  Research  Yarn  and  Other  Comments." 

JOHN  W.  TUKEY,  "The  Future  of  Processes  of  Data  Analysis." 

M.  G.  KENDALL,  "Statistics  and  Management." 

11th:  Dover,  New  Jersey,  20-22  October  1965. 

Headquarters,  U.S.  Army  Munitions  Command. 

JOAN  R.  ROSENBLATT,  "Confidence  Limits  for  the  Reliability  of 
Complex  Systems . " 

J.  STUART  HUNTER,  "Non-Linear  Models:  Estimation  and  Design." 

WILLIAM  C.  GUENTHER,  "Target  Coverage  Problems." 

H.  0.  HARTLEY,  "Maximum  Likelihood  Estimates  for  the  General 
Mixed  Analysis  of  a Variance  Model." 

12th;  Washington,  D.C.,  19-21  October  1966. 

Harry  Diamond  Laboratories  and  National  Bureau  of  Standards. 

BRIAN  W.  C0N0LLY,  "Operations  Research." 

JOHN  MANDEL,  "Statistics  as  a Diagnostic  Tool  in  Data  Analysis." 

W,  G.  COCHRAN,  "Planning  and  Analysis  of  Observational  Studies." 

NORMAN  L.  JOHNSON,  "Sample  Censoring." 

13th : Fort  Belvoir,  Virginia,  1“3  November  1967. 

U.S.  Army  Mobility  Equipment  Center  and  U.S.  Army  Engineer  Topographic 
Laboratories . 

FRANCIS  J.  ANSCOMBE,  "Regression  Analysis." 

K.  A.  BROWNLEE,  "Some  Comments  on  Matching." 

I.  J.  GOOD,  "Some  Statistical  Methods  in  Machine  Intelligence  Research." 
FRANK  PROSCHAN,  "Maximum  Likelihood  Estimation  of  Reliability." 


M.  B,  WILK,  "Data  Analysis.” 

14th:  Edgewood  Arsenal,  Maryland  23-25  October  1968. 

U.S.  Army  Edgewood  Arsenal, 

LT.  GEN.  WILLIAM  B.  BUNKER,  "Broadening  the  Horizons  of 
Experimental  Design." 

ROLF  E.  BARGMANN,  "Structure  and  Classification  of  Patterns." 

ACHESON  J.  DUNCAN,  "Bulk  Sampling." 

EMANUEL  PARZEN,  "Time  Series." 

15th:  Redstone  Arsenal,  Alabama,  22-24  October  1969. 

U.S.  Army  Missile  Command. 

JOHN  E.  CONDON,  "Reliability  and  Quality  Assurance," 

NANCY  R.  MANN,  "Systems  Reliability." 

CLIFFORD  J.  MALONEY,  "A  Probability  Approach  to  Catastrophic 
Threat 

RICHARD  G.  KRUTCHKOFF,  "The  Empirical  Bayes  Approach  to  the  Design 
and  Analysis  of  Experiments." 

S.  C.  SAUNDERS,  "On  Confidence  Limits  for  the  Performance  of  a 
System  When  Few  Failures  are  Encountered," 

16th:  Fort  Lee,  Virginia,  21-23  October  1970. 

U.S.  Army  Logistics  Management  Center. 

SOLOMON  KULLBACK,  "Minimum  Discrimination  Information  Estimation 
and  Application." 

RICHARD  J.  KAPLAN,  "Field  Testing," 

GARY  G.  KOCH,  "The  Analysis  of  Complex  Contingency  Table  Data 
from  General  Experimental  Designs  and  Sample  Surveys." 

A.  CLIFFORD  COHEN,  "Estimation  in  Truncated  Poisson  Distributions 
with  Concomitant  Intervals  and  Truncation  Points." 

DANA  QUADE,  "Nonparametric  Analysis  of  Covariance." 
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17th:  Washington,  D.C.,  27-29  October  1971. 

Walter  Reed  Army  Institute  of  Research,  Walter  Reed  Army  Medical 
Center. 

MARVIN  ZELEN,  "The  Role  of  Mathematical  Sciences  in  Biomedical 
Research." 

BERNARD  G.  GREENBERG,  "Randomized  Response:  A New  Survey  Tool  to 
Collect  Data  of  a Personal  Nature." 

GEOFFREY  H.  BALL,  "Classification  and  Clustering  Techniques  in 
Data  Analysis." 

K.  S,  BANERJEE,  "Hotelling's  Weighing  Designs." 

JOHN  J.  GART,  "The  Comparison  of  Proportions:  A Review  of 
Significance  Teats,  Confidence  Intervals  and  Adjustments  for 
Stratification. " 

18th : Aberdeen  Proving  Ground,  Maryland,  25-27  October  1972. 

U.S.  Army  Test  and  Evaluation  Command. 

JOHN  TUKEY,  "Exploratory  Data  Analysis." 

G.  S.  WATSON,  "Orientation  Analysis." 

J.  STUART  HUNTER,  "Sequential  Factorial  Estimation." 

G,  E.  P.  BOX, "Forecasting  and  Control." 

RAYMOND  H.  MYERS,  "Dual  Response  Surface  Analysis," 

19th : Rock  Island,  Illinois,  24-26  October  1973. 

Headquarters  U.S.  Army  Armament  Command  and  U.S,  Army  Management 
Engineer  Training  Agency. 

JEROME  CORNFIELD,  "Bayesian  Statistics." 

SHANTI  GUPTA,  "Ranking  Based  on  Multiple  Criteria." 

J.  SETHURAMAN,  "A  Strong  Justification  for  the  Use  of  Rank  Tests 
in  the  Case  of  Non-Normality." 

H.  L.  GRAY,  "Generalized  Jackknife  Techniques." 

FRANK  PRQSCHAN,  "Reliability  Growth." 

SAM  C.  SAUNDERS,  "Accelerated  Life  Testing." 

WILLIAM  A,  THOMPSON,  JR.,  "Reliability  of  Multiple  Component  Systems." 
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Solomon  Kullback 
The  Florida  State  University 
Department  of  Statistics 

ABSTRACT . Through  the  use  of  the  principle  of  minimum  discrimination 
information  estimation,  leading  to  exponential  families  or  multiplicative 
models  or  log-linear  models  it  has  been  shown,  using  illustrative  examp- 
les exhibiting  different  aspects  of  contingency  table  analysis,  that. 

(1)  Estimates  of  the  cell  entries  under  various  hypotheses 
or  models  can  be  obtained; 

(2)  The  adequacy  or  fit  of  the  model,  or  the  null  hypothesis, 
can  be  tested; 

(3)  Main  effect  and  interaction  parameters  can  be  estimated; 

( k ) The  structure  of  the  table  can  be  studied  in  detail  in 

terms  of  the  various  interrelationships  among  the  classi- 
f.icatory  variables; 

(5)  The  procedures  can  be  applied  to  test  hypotheses  about 
particular  parameters  and  linear  combinations  of  parameters 
that  are  of  special  interest; 

(6)  The  procedures  provide  indication  of  outlier  cells; 

(7)  Since  the  procedures  and  concepts  are  based  on  a general 
principle  a unified  treatment  of  multi-dimensional  contin- 
gency tables  is  possible; 

(8)  The  procedure  provides  estimates  based  on  an  observed  or 
sample  table,  which  satisfy  certain  external  hypotheses 
as  to  underlying  probability  relations  in  the  population 
table.  These  estimates  also  preserve  the  inherent  pro- 
perties of  the  observed  data  not  affected  by  the  hypo- 
thesis; 

(9)  In  general,  the  m.d.i.  estimate  is  best  asymptotically 
normal ; 

(10)  The  minimum  discrimination  information  test  statistics  are 
asymptotically  distributed  as  chi-squared  with  appropriate 
degrees  of  freedom; 

(11)  Convergent  iterative  computer  algorithms  are  available 
for  the  analyses . 

COKTlJffGKjflCY  TA3LB3 . There  are  two  ways  in  which  Statistical  data  are 
collected.  In  one  form,  actual  measurements  are  recorded  for  each 
individual  in  the  sample;  in  the  other,  the  individuals  are  classi- 
fied as  belonging  to  different  categories.  On  many  occasions  classi- 
fications are  used  to  reduce  original  data  on  direct  measurements , A 
well  known  example  is  that  of  M frequency-distributions”.  Data  collec- 
ted in  the  form  of  measurements  may  later  be  grouped  and  presented  as 
a frequency  distribution.  An  important  advantage  of  grouping  is  that 
it  results  in  a considerable  reduction  of  data.  On  the  other  hand,  it 
is  not  usually  possible  to  convert  grouped  or  classified  data  back  into 
the  original  form. 

Data  which  results  from  experiments  in  the  physical  sciences  and 
engineering  are  usually  outcomes  of  controlled  experiments,  and  expres- 


sible  in  quantitative  terms.  In  many  other  fields  however , the  data 
are  seldom  results  of  controlled  experiments.  In  addition,  the  ob— 
servations  usually  can  he  expressed  only  in  qualitative  or  categorical 
terms,  a yes  - no,  alive  - dead,  agree  - disagree,  class  A - class 
B - class  C,  etc.  type  of  response. 

A contingency  table  is  a form  of  presentation  of  grouped  data.  In 
the  simplest  case,  a group  of  N items  may  be  classified  into  just  two 
groups,  according  to,  say,  presence  or  absence  of  a certain  characteristic. 
For  a fixed  (given)  characteristic  the  different  groups  of  classification 
are  called  categories . For  example , a group  of  N individuals  may  be 
classified  according  to  hair  color  (characteristic),  the  categories  being 
black,  brown,  blonde  and  "other".  The  categories  may  be  qualitative  as 
above,  or  may  be  quantitative,  as  for  example  in  the  classification  by 
weight  in  pounds  consisting  of  five  categories:  H0-80,  80-120,  120-160, 
160-200,  200-2^0 . When  there  is  only  one  characteristic  according  to 
which  data  are  classified  we  get  a one-way-table . If  there  are  two 
ways  of  classification,  say  according  to  Hows  and  Columns,  the  Row- 
classification  having  r categories  and  the  Column-classification  having 
c categories , the  table  is  called  a two-way  table  or  a r x c table . 

The  latter  notation  gives  the  number  of  categories  in  each  classification. 
Carrying  this  notation  further,  a r x c x d table  will  have  three  char- 
acteristics of  classification,  the  first  having  r categories,  the  second 
having  c and  the  third  d. 

For  example,  an  individual  may  be  classified  by  sex,  by  race,  by 
profession,  by  smoking  habit,  by  age,  by  incidence  of  coronary  heart 
disease.  If  we  take  observations  over  a sample  of  many  such  individuals, 
the  result  will  he  a multidimensional  contingency  table  with  as  many 
dimensions  as  there  are  classifications.  Contingency  tables  are  cross- 
classifications of  vectors  of  discrete  random  variables  showing  the  num- 
ber of  subjects  belonging  to  distinct  categories  of  each  of  several  quali- 
tative or  categorical  classifications.  The  number  of  counts  of  individuals 
in  a cell  of  this  table  represents  that  portion  of  the  sample  having  the  spe 
cific  attributes  within  each  of  the  classifications.  A problem  of  interest, 
for  example,  might  be  to  determine  the  factors  that  are  associated  with  the 
presence  or  absence  of  coronary  heart  disease. 

Data  from  many  fields  are  often  presented  in  this  manner,  that  is, 
in  a cross-tabulated  form.  Statistical  analyses  of  these  types  of  data 
has  had  a long  history,  but  were  mainly  concerned  with  the  simple  kind, 
the  two-way  table.  Analyses  of  multidimensional  contingency  tables  have 
been  investigated  intensively  only  during  the  last  decade  or  so. 

Conclusions  drawn  from  contingency  tables  may  be  only  exploratory 
in  nature.  One  of  the  difficulties  can  be  the  availability  of  meaning- 
ful and  reliable  data.  The  first  problem  one  faces  in  the  analysis  of 
cross-classified  data  is  the  decision  on  the  number  of  classifications 
to  be  included  and  the  categories  within  each  classification.  Typical 
among  the  problems  in  the  analysis  is  how  to  segregate  the  effect  on 
the  response  of  some  of  the  background  variables,  individually  or  jointly, 
from  that  of  the  others  that  are  of  particular  interest.  The  data  analy- 
tic attitude  is  empirical  rather  than  theoretical.  A more  empirical 
attitude  is  natural  when  detailed  theoretical  understanding  is  unavailable  , 
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Estimation  of  parameters  in  models  should  be  considered  less  as  attempts 
to  discover  underlying  truths  and  more  as  data  calibrating  devices  which 
make  it  easier  to  conceive  of  noisy  data  in  terms  of  smooth  distn  u ions 
and  relations.  With  a given  data  set,  a variety  of  models  may  be  trie 
on,  and  one  selected  on  the  ground  oi  looks  and  fit. 

In  the  analysis  of  contingency  tables  ve  are  usually  interested  in 
the  relationship  between  one  classification  and  one  or  more  of  the  other 
classifications.  As  an  example,  consider  a three-way  r x c x d contin- 
gency table  in  which  the  row-classification  represents  the  response  of 
an  experiment  on  animals,  the  column  classification  types  of  treatment 
and  the  depth  classification  sex.  The  following  hypotheses  may  be  of 
interest . 

1.  Response  is  independent  of  treatment  irrespective  of  sex. 

2.  Response  is  independent  of  the  different  combinations  of 
treatment  and  sex  (as  against  the  possibility  that  a parti- 
cular treatment  is  more  "effective’1  in  terms  of  the  response , 
for  a particular  sex), 

3.  Given  sex,  response  is  independent  of  treatment. 

Of  course,  not  all  contingency  tables  can  be  interpreted  in  such 
a straightforward  manner.  In  some  instances,  all  three  classifications 
can  be  considered  as  responses;  then  we  may  be  interested  in  the  inde- 
pendence or  association  among  these  responses.  In  other  cases,  a classi- 
fication may  be  controlled,  experimentally  or  naturally,  like  three  speci- 
fied levels  of  fertilizer  applied  or  sex,  and  then  the  classification  is 
termed  a factor.  For  convenience,  we  shall  group  all  the  concepts  of 
association,  dependence,  etc.  under  the  general  term  of  interaction, 
ho  interaction  between  treatment  and  sex  appears  to  be  a more  acceptable 
phrase  than  independence  between  treatment  and  sex,  since  the  term  inde- 
pendence is  usually  reserved  to  express  the  relationship  between  random 
variables . We  may  also  say  that  the  interaction  between  response  and 
treatment  does  not  interact  with  sex,  meaning  the  degree  of  association 
between  response  and  treatment  is  the  same  for  both  sexes.  The  concept 
gives  rise  to  the  idea  of  second-order  interaction.  There  are  a number 
of  different  approaches  to  the  mathematical  formulation  and  interpretation 
of  the  concept  of  "no  interaction”.  One  such  approach,  through  the  con- 
cept of  ’’generalized  independence”  is  powerful  and  general  enough  to  in- 
clude all  hypotheses  of  "no  interaction”  (formulated  in  a specific  manner) 
and  many  other  hypotheses  about  homogeneity,  symmetry,  etc.  that  we  come 
across  in  analysing  contingency  tables. 

Consider,  for  example,  an  experiment  to  compare  the  effectiveness  of 
safety  release  devices  for  regrigerators  in  relation  to  children's  safety. 
Children  between  two  to  five  years  of  age  are  induced  to  crawl  into  refri- 
gerators equipped  with  six  different  types  of  release  devices . If  a child 
can  open  the  door  of  the  refrigerator,  from  inside,  within  a certain  time 
period,  the  response  is  classified  as  a success,  otherwise  a failure.  The 
background  variables  studied  included  age,  sex,  weight,  socio-economic 
status  of  parents . The  experimental  variable  was  one  of  six  devices . 

(A  partial  analysis  of  this  data  may  be  found  in  page  581  of  Kullback,  S. , 
Kupperman,  M. , and  Ku,  H.H,(1962),  Tests  for  contingency  tables  and  Markov 
chains,  Technometrics , 4,  573- 608).  Some  balancing  of  the  background  vari- 
ables was  achieved. 
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In  other  instances  none  of  the  factors  are  subject  to  experimental 
control,  and  whatever  available  data  could  be  collected  is  reported. 

The  analysis  of  this  type  of  data,  though  it  may  Only  be  seeking  prelim- 
nary  information  can  be  important  in  fields  of  health  and  safety.  The 
uncontrolled  experimental  data  are  sometimes  the  only  realistic  data 
available  when  these  data  deal  with  life,  death,  health,  and  safety,  and 
some  of  these  factors  and  responses  are  only  expressible  in  qualitative 
terms,  in  the  present  state  of  art. 

It  is  expected  that  the  number  of  problems  calling  for  the  tech- 
niques of  the  analysis  of  multidimensional  contingency  tables  will 
increase.  Experience  at  the  George  Washington  University  with  such  a 
growing  demand  confirms  this.  The  examination  and  interpretation  of 
data  from  social  phenomena,  housing,  psychology,  education,  environmen- 
tal problems,  health,  safety,  manpower,  business,  experimental  testing 
of  devices,  military  research  and  development,  etc.,  are  potential  source 
areas . 

Classical  problems  in  the  historical  development  of  the  analysis  of 
contingency  tables  concerned  themselves  primarily  with  such  questions  as 
the  independence  or  conditional  independence  of  the  classificatory  vari- 
ables, or  homogeneity  or  conditional  homogeneity  of  the  classificatory 
Variables . over  time  or  space,  for  example,  similar  to  such  tests  in  multi- 
variate analysis  as  independance , multiple  correlation,  partial  correlation, 
canonical  correlation,  etc.  Such  classical  problems  turn  out  to  be  special 
cases  of  the  technique  we  discuss . 

These  techniques  result  in  analyses  which  are  essentially  regression 
type  analyses . As  such  they  enable  us  to  determine  the  relationship  of 
one  or  more  "dependent"  qualitative  or  categorical  variables  of  interest 
on  a set  of  "independent"  classificatory  variables,  as  well  as  the  rela- 
tive effects  of  changes  in  the  "independent"  variables  on  the  "dependent 
variables".  The  object  of  the  analyses  is  the  study  of  the  interaction 
between  and  among  the  classifications.  The  term  interaction  is  used  here 
in  a general  sense  to  cover  both  dependence  and  association. 

Critics  of  methods  for  contingency  table  analysis  have  maintained 
that  most  of  the  procedures  used,  at  least  in  the  past,  were  only  of  a 
global  chi-squared  test  nature.  However,  for  a recent  example  of  this 
see  Patil,  K.D,  (197^ ^Interaction  test  for  three-dimensional  contingency 
tables.  Journal  Am.  Statist.  Assn.  6^>  l6t— 168.  Through  the  use  of  the 
principle  of  minimum  discrimination  information: (m.d.i . ) estimation,  lead- 
ing to  exponential  families  or  multiplicative  models  (generalized  indepen- 
dence) or  log-linear  models  we  show  that: 

(1)  Estimates  of  the  cell  entries  under  various  hypotheses  or  models 
can  be  obtained; 

(2)  The  adequacy  or  fit  of  the  model,  or  the  null  hypothesis,  can 
be  tested; 

(3)  Main  effect  and  interaction  parameters  can  be  estimated; 

( ) structure  of  the  table  can  be  studied  in  detail  in  terms 
of  the  various  interrelationships  among  the  classificatory 
variables; 

(5)  The  procedures  can  be  applied  to  test  hypotheses  about  particur 
lar  parameters  and  linear  combinations  of  parameters  that  are 
of  special  interest; 
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(6)  The  procedures  provide  indication  of  outlier  cells.  These  may 
cause  a model  not  to  fit  overall,  yet  fit  the  other  cells  exclu- 
ding  the  outliers; 

(7)  Since  the  procedures  and  concepts  are  based  on  a general  prin- 
ciple a unified  treatment  of  multidimensional  contingency  tables 
is  possible.  Sequences  of  generalizations  step  by  step  to  high- 
er order  dimensional  contingency  tables  are  not  necessary  as  has 
been  the  case  with  other  ad  hoc  procedures  (see  for  example, 

Patil  (197*0  * Sugiura,  N and  Otake , H.  (197*+),  An  extension  of 
Mantel-Haens  zel  procedure  to  k 2 x c contingency  tables  and  the 
relation  to  the  logit  model,  Communications  in  Statistics,  to 
appear ) ; 

(8)  The  procedure  provides  estimates  based  on  an  observed  or  sample 
table,  which  satisfy  certain  external  hypotheses  as  to  under- 
lying probability  relations  in  the  population  table.  These 
estimates  also  preserve  the  inherent  properties  of  the  observed 
data  not  affected  by  the  hypothesis; 

(9)  In  general,  the  m.d.i.  estimates  are  best  asymptotically  normal 
(BAN)  and  in  the  many  applications  of  fitting  models  to  a table 
based  on  observed  sets  of  marginal  values  or  linear  restraints 
of  observed  values,  the  m.d.i.  estimates  in  particular  are 
maximum-likelihood  estimates; 

(10)  Hie  test  statistics  are  minimum  discrimination  information 

( m.d.i.)  statistics  which  are  asymptotically  distributed  as 
chi-squared  with  appropriate  degrees  of  freedom.  In  the  case 
of  fitting  models  to  a table  based  on  observed  sets  of  marginal 
values  or  linear  restraints  of  observed  values,  the  m.d.i.  stat- 
istics are  log-likelihood  ratio  statistics.  The  m.d.i,  statis- 
tics are  additive,  as  are  the  associated  degrees  of  freedom,  so 
that  the  total  under  an  hypothesis  can  be  analyzed  into  components 
each  under  sub-hypotheses.  The  analysis  is  analogous  to  analy- 
sis of  variance  and  regression  analysis  techniques.  It  uses  a 
design  matrix,  a set  of  regression  parameters,  and  explanatory 
variables,  and  analysis  of  information  tables. 

(11)  In  models  fitting  estimates  to  an  observed  table  based  on  sets  of 
observed  marginal  values  as  explanatory  variables,  some  estimates 
can  be  expressed  explicitly  as  products  of  marginal  values. 

However,  this  is  not  generally  true,  and  expected  cell  frequencies 
(functions  of  marginal  values),  can  be  computed  by  an  iterative 
proportional  fitting  procedure,  and  the  use  of  a computer  to  perform 
the  iterations  becomes  necessary.  For  the  foregoing  cases* which  we 
term  internal,  and  problems  involving  tests  of  external  hypotheses 
on  underlying  populations  a number  of  iterative  computer  programs 
are  available.  They  provide  as  output,  design  matrices,  the  obser- 
ved cell  entries  and  the  cell  estimates  as  well  as  their  logarithms, 
parameter  estimates,  outlier  values,  m.d.i.  statistics  and  their 
corresponding  significance  levels,  and  covariance  matrices  of  para- 
meter estimates,  to  assist  in  and  simplify  the  numerical  aspects  of 
the  inference.  In  this  respect  it  is  of  interest  to  cite  the 
following  quotation  from  a book  review  by  D.J.  Finney  in  Journal 
Royal  Statistical  Society,  Series  A(General)  Vol.  136(1973)  , part 

3 , p*  *+6l,  '*No  mention- is  made  of  the  extent  to  which  computers 
have  destroyed  the  need  to  assess  statistical  methods  in  terms  of 
arithmetical,  simplicity:  indeed  the  emphasis  on  avoiding  lengthy, 

but  easily  programmed,  iterative  calculations  is  remarkable”. 
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ABSTRACT , The  simulation  of  radar  scattering  signatures,  including 
radar  cross  section  and  glint,  of  complex  targets  for  use  in  air  defense 
system  simulations  is  a difficult  and  time  consuming  task.  Although  it 
is  possible  to  develop  deterministic  models  of  the  radar  signature,  as  a 
function  of  the  target  aspect  angles,  it  is  generally  not  possible  to  use 
these  models  in  a realtime  simulation  because  of  the  computational  require- 
ments involved  in  using  such  a model.  Statistical  and  stochastical  models 
of  radar  signatures  are  generally  limited  to  the  classical  radar  cross 
section  models,  although  some  models  do  include  crudely  correlated  glint 
models  as  well.  It  is  possible  to  describe  statistically  the  radar  scat- 
tering signature  in  terms  of  the  probability  density  and  covariance  func- 
tions, but  the  processes  generally  are  non- stationary,  non-Gaussian,  non- 
Markovian  processes.  Even  reduction  of  the  process  to  a stationary, 
Markovian,  non-Gaussian  process  does  not  presently  reduce  the  problem  to 
an  analytically  solvable  problem.  The  development  of  techniques  to  gener- 
ate these  multidimensional  random  processes  is  needed  to  make  more  real- 
istic simulations  practical. 


I.  INTRODUCTION.  The  simulation  of  realistic  radar  signatures  of 
aircraft  for  use  in  air  defense  (AD)  system  simulations  is  a difficult 
task.  Stochastic  models  capable  of  representing  the  multidimensional 
radar  signature  with  any  realism  do  not  exist,  so  complex  deterministic 
models  are  used  when  realism  is  required.  These  deterministic  models 
require  significant  computer  resources  in  terms  of  both  computation  time 
and  memory,  but  they  can  represent  the  nonstationary  multidimensional 
signature  with  sufficient  accuracy  to  make  them  invaluable  in  all  digital 
simulations.  The  computation  time  requirements  generally  exclude  deter- 
ministic models  from  realtime  hybrid  simulations,  i.e.,  those  simulations 
with  actual  system  hardware  in  the  loop.  It  is  for  these  realtime,  hybrid 
simulations  that  realistic  stochastic  models  are  needed. 

The  purpose  of  this  paper  is  to  present  the  problem  with  a descrip- 
tion of  the  process  and  the  underlying  phenomena  from  which  it  is  derived. 


55 


II.  DESCRIPTION  OF  THE  PROBLEM 


1.  General . The  first  step  is  to  establish  the  definition  and 
description  of  the  radar  signature.  The  elements,  or  parameters,  of  the 
signature  can  be  addressed  one  at  a time  or  in  pairs  to  arrive  at  a 
statistical  description  of  the  overall  process. 

The  radar  signature  is  defined  to  be  the  set  of  target  induced  param- 
eters which  are  measured  by  the  observing  radar(s)  or  which  directly 
influence  the  radar  measurements  or  tracking  systems.  These  include  the 
radar  cross  section  (RCS) , azimuth  glint  (e^ ) , elevation  glint  (ef  ) , and 

intrinsic  phase  (a).  Other  parameters,  such  as  range  glint,  which  are 
dependent  on  system  mechanization  will  not  be  considered.  Each  of  these 
four  parameters  is  range  independent  for  far  field  conditions,  which  is 
assumed  for  simplicity.  Since  most  of  the  Army  AD  systems  are  semi-active 
systems,  each  of  these  parameters  must  be  considered  twice:  once  for  the 

ground  radar,  and  once  for  the  missile  seeker.  The  signature,  therefore, 
is  eight  dimensional. 

The  problem  is  further  complicated  by  the  fact  that  these  parameters 
are  functions  of  the  target  aspect  angles  which  are  nonstationary  functions 
of  time.  Figure  1 depicts  the  plan  view  of  an  arbitrarily  selected  flight 
path  and  Figure  2 depicts  the  aspect  angles  of  a perfectly  controlled  tar- 
get in  still  air  as  seen  from  a ground  radar  located  at  the  origin.  (The 
+ sysbols  are  taken  at  equal  time  intervals.)  A realistic  target  will 
experience  random  rotational  (and  resulting  translational)  perturbations 
from  wind  gusts  and  autopilot  noise  and  response  characteristics,  so  that 
the  actual  aspect  angles  will  be  a two-di:nensional  random  process  with 
averages  approximately  as  shown  in  Figure  2.  The  nature  of  the  random 
perturbations  is  a function  of  the  assumed  environment  and  the  target 
aerodynamic  and  control  response  characteristics. 

The  problem,  then,  is  to  develop  a stochastic  model  of  a nonstationary 
eight-dimensional  random  process.  The  correlations  among  the  various  param- 
eters must  be  maintained  because  they  significantly  affect  the  response  of 
the  AD  system.  It  is  clear  that  some  simplifications  to  the  problem  are 
required  before  any  attack  can  be  made  on  the  problem.  An  investigation  of 
the  various  parts  of  the  problem  will  reveal  some  significant  simplifications 

2.  Target  Signature  Characteristics.  The  most  realistic  deter- 
ministic radar  signature  models  are  based  on  the  N-body  approach.  This 
approach  assumes  that  the  signature  can  be  assumed  to  be  generated  by  N 
scatterers  or  scattering  centers.  The  components  of  the  radar  signature 
can  be  given  by  [ 1] : 

N N 
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RCS  = 


is  the  RCS  of  the  i-th  scatterer, 

and 

a is  the  phase  angle  associated  with  the  i-th  scatterer. 

The  RCS  is  given  in  square  meters,  and  the  eQ  and  e are  given  as 

errors  in  the  apparent  target  position,  in  meters,  in  a plane  orthogonal 
to  the  line  of  sight  at  the  target.  It  has  been  demonstrated  that  the 

glint  errors,  e„  and  e , can  be  expressed  as  the  gradients  of  the  phase 

0 cp 

with  respect  to  the  appropriate  angles  [2,3].  Conversely,  it  is  possible 
to  compute  the  random  component  of  the  intrinsic  phase  as  the  integral  of 
the  phase  gradient. 

Figures  3,4,  and  5 depict  the  RCS,  RCS*  eQ  (theta  component  of  non- 

radial  power)  and  RCS*  (phi  component  of  non-radial  power)  for  a 

selected  region  of  aspect  angles  for  a simple  mathematical  model  of  the 
MQM-34D  (BQM-34A)  target  drone  [1]  for  a frequency  of  1 GHz.  The  non- 
linear nature  of  these  functions  is  clear,  but  the  correlation  is  not. 
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The  correlation  of  interest  is  the  statistical  correlation.  If  it  is 
assumed  that  the  aspect  angles  have  some  statistical  relationships,  it 
is  possible  to  compute  the  statistical  relationships  of  the  scattering 
components . 

The  nonstationary  nature  of  the  process  can  be  handled  in  a relatively 
straightforward  manner.  Since  the  average  aspect  angles  generally  change 
slowly  with  respect  to  the  random  components,  it  is  reasonable  to  consider 
the  random  component  separately.  It  is  assumed  that  the  random  components 
of  the  aspect  angles  are  stationary  Markov  processes.  This  is  not  strictly 
true  but  the  lack  of  definitive  measurements  on  aircraft  motion  and  the 
realism  of  the  radar  signature  models  make  the  assumption  acceptable.  Since 
the  statistical  characteristics  of  the  signature  components  are  functions 
of  the  aspect  angles,  the  statistical  characteristics  of  the  signature  are 
also  assumed  to  be  stationary  processes  when  considered  from  the  short  term 
viewpoint . 


3.  Statistical  Characteristics  of  the  Radar  Signature.  The  statis- 
tical characteristics  of  the  radar  signature  of  a target  are  functions  of 
many  variables,  two  especially  important  ones  being  radar  frequency  and 
aspect  angle  statistics.  Figures  3,  4,  and  5 were  computed  for  1 GHz. 

The  lobing  structure  increases  approximately  linearly  with  frequency,  but 
the  averages  and  variances  do  not.  A Monte  Carlo  type  simulation  of  the 
target  at  a point  corresponding  to  6 = 103.8  degrees,  cp  = 38.67  degrees 
with  6 and  cp  jointly  normal  and  cr^  - 3.07  degrees  a,  = 1.46  degrees,  and 


= 0.543  resulted  in  the  results  shown  in  Figures  6 through  11.  These 

figures  present  plots  of  the  radar  signature  parameters  or  related  param- 
eters. These  plots  indicate  the  type  of  random  processes  that  are  to  be 
modeled.  Figures  12  through  16  present  probability  density  functions  of 
the  data  in  Figures  6 through  11  and  Figures  17  through  20  present  typical 
covariance  functions  for  part  of  these  data. 


Three  classical  statistical  RCS  models  are  of  interest.  They  are  the 
Swerling  1,  Swerling  3,  and  log-normal.  The  equations  for  these  models 
are  given  by: 


Swerling  1 


where 


Swerling  3 


*s  = E {s}  , 


fs(s)  = exp  /-  fr 


s > 0 


s > 0 
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and  log-normal 


f (s) 
s v 7 


as  /ET 


exp 


(in  s ■ [i) 
2a2 


s > 0 


where 

jj.  = E {in  s} 

and 


a2  = E { (in  s - p) 2] 

Models  of  the  nonradial  components  of  power  have  not  been  developed  at 
this  time. 

The  problem  of  bistatic  angles  must  also  be  addressed.  It  can  be 
shown  that  the  bistatic  signature  is  best  approximated  by  the  monostatic 
signature  for  the  aspect  angles  corresponding  to  those  of  the  bisector 
of  the  ground  radar  aspect  angles  and  the  missile  seeker  aspect  angles 
reduced  by  a scale  factor  which  is  a function  of  the  bistatic  angle.  The 
covariance  of  the  RCS  as  a function  of  one  half  of  the  bistatic  angle  is 
approximately  the  same  shape  as  the  covariance  of  the  RCS  as  a function 
of  the  aspect  angle. 

The  roll-off  of  RCS  as  a function  of  bistatic  angle  for  the  MQM-34D 
drone  near  nose-on  appears  to  be  exponential  in  shape  with  a reduction  of 
approximately  4.5  decibels  (multiplicative  factor  of  approximately  3)  in 
30  degrees  in  average  and  standard  deviation, 

A review  of  the  known  data  is  appropriate  at  this  point  to  determine 
what  data  are  missing.  The  first  order  probability  density  functions  and 
covariance  functions  can  be  assumed  to  be  known  for  the  monostatic  RCS 
and  nonradial  components  of  power  and  for  the  bistatic  RCS  and  nonradial 
components  of  power.  The  covariance  function  of  the  monostatic  and 
bistatic  RCS  is  also  known.  The  covariance  functions  of  the  monostatic 
and  bistatic  nonradial  components  of  power  are  not  known,  but  theory  and 
experiments  indicate  that  for  the  frequencies  of  interest  and  bistatic 
angles  exceeding  5 degrees,  these  covariance  functions  can  be  assumed  to 
be  zero.  The  intrinsic  phase  is  most  readily  computed  as  the  integral  of 
the  glint  (phase  gradient)  since  absolute  phase  is  not  important.  Thus 
the  stochastic  process  is  actually  reduced  basically  to  a pair  of  three- 
dimensional  random  processes,  with  a correlation  between  the  monostatic 
and  bistatic  RCS  for  small  bistatic  angles. 

The  first  order  probability  density  functions  for  the  RCS  will 
generally  be  selected  from  one  of  the  three  classical  models  previously 
given.  The  probability  density  functions  for  the  nonradial  components 
of  power  remain  to  be  determined,  at  least  in  analytical  form.  The 
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covariance  functions  appear  to  be  approximately  exponential  for  the  cases 
studied  to  date.  Analysis  of  other  deterministic  models,  the  MQM-34D 
and  other  targets,  may  indicate  other  shapes  for  the  covariance  functions, 
however. 

The  nonstationary  aspect  of  the  problem  can  probably  be  handled  by 
using  averages  and  variances  which  are  functions  of  the  aspect  angles  and, 
hence,  of  time. 

That  leaves  one  major  problem  area,  the  generation  of  a three-dimen' 
sional  random  process  with  non-normal  probability  density  functions  and 
specified  covariance  functions.  Techniques  are  available  for  handling 
limited  classes  of  one -dimensional  random  processes  [4,  5,  6]  , but  it 
appears  that  generalization  to  multidimensional  processes,  except  for 
very  special  cases  [ 5] , has  not  been  accomplished  [ 7]  . 


III.  CONCLUSION.  This  paper  has  outlined  one  area  where  the  multi- 
dimensional random  processes  are  needed  today.  No  attempt  has  been  made 
to  present  all  of  the  data  necessary  to  completely  define  the  problem. 

In  fact,  the  author  is  not  sure  what  data  are  needed  to  completely  define 
the  nature  of  the  stochastic  process  that  is  to  be  modeled.  The  data 
presented  are  generally  accepted  as  necessary  but  are  probably  not  suf- 
ficient to  permit  complete  characterization. 
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Figure  1.  Plan  view  of  target  flight  path. 
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DESIGN  OF  EXPERIMENTS  FOR  THE  EVALUATION  OF 
MATERIEL  PERFORMANCE  IN  WORLDWIDE  ENVIRONMENTS 


Bob  0,  Benn 

Waterways  Experiment  Station 
Corps  of  Engineers 
Vicksburg,  Mississippi 

Background 

A review  of  recently  approved  Required  Operational  Capability 
(ROC)  documents  reveals  that  the  user  in  the  Army  is  requesting  a 
host  of  materiel  items  with  truly  exceptional  performance  capabilities. 
A good  example  of  this  trend  toward  materiel  sophistication  can  be 
found  in  the  ROC's  dealing  with  intrusion  detection,  target-position 
location,  and  target  discrimination.  It  is  axiomatic  that  the  more 
complex  the  system,  the  more  sensitive  it  can  be  to  its  operational 
environment.  Nevertheless,  the  systems  are  intended  to  function 
adequately  in  the  majority  of  worldwide  terrain  or  environmental 
conditions. 

The  Test  Methodology  Directorate,  U.  S.  Amy  Test  and  Evaluation 
Command  (TECOM),  has  recognized  for  some  time  that  adequate  testing 
procedures  are  not  available  for  comprehensive  evaluation  of  materiel 
of  the  type  discussed  in  the  preceding  paragraph.  To  a large  extent, 
the  inadequacy  results  because  empirical  tests  to  evaluate  materiel 
items  are  conducted  in  a specific  (and  only  a limited  number  of) 
terrain  or  environmental  conditions;  yet  those  test  results  must  be 
extrapolated  to  worldwide  conditions  if  the  evaluations  are  to  be" 
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conclusive.  As  part  of  TECOM's  endeavor  to  improve  its  test  and  evaluation 
capability,  the  U.  S,  Army  Engineer  Waterways  Experiment  Station  (WES) 
was  asked  to  develop  test  guidance  and  analytical  procedures  that  could 
be  used  to  extrapolate  to  worldwide  environments  the  results  obtained  in 
tests  with  unattended  sensors  and  mines  in  specific  terrains. 

In  general,  the  problem  to  be  addressed  was  the  development 
of  guidelines  for  designing  an  experiment  in  which  the  items  could  be 
evaluated  to  determine  if  they  will  function  above  the  minimum  operating 
criteria  stated  in  the  ROC.  Conventionally,  the  chief  objectives  of 
experimental  design  are  to: 

a,  Arrange  the  experiment  so  that  the  effects  of  changing 

each  relevant  condition  or  factor  can  be  readily  measured 
independent  of  the  effects  of  changing  the  other  condi- 
tions and  of  experimental  error. 

_b.  Obtain  a valid  estimate  of  error  appropriate  for  assessing 

independently  or  synergist ically  the  statistical  significance 
of  the  effects  of  the  factors  considered, 
c^.  Enable  the  effects  to  he  measured  with  the  required  accuracy. 

Normally,  the  experiment  is  arranged  so  that  one  (or  at  most  two 
or  three)  of  the  factors  or  conditions  which  are  known  to  be  significant 
are  varied  incrementally  while  all  others  are  held  constant.  This 
permits  the  effects  of  those  factors  to  be  determined,  but  only  as 
independent  variables.  Such  a procedure  does  not  permit  the  evaluation 
of  the  effects  of  all  of  the  relevant  factors  acting  in  concert,  yet 
that  is  invariably  the  way  they  act  when  the  device  is  in  operational 
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use.  Nor  does  this  procedure  make  any  provision  for  the  recognition  of 
the  factors  which  are  not  known  to  be  significant  at  the  beginning  of 
the  experiment  or  test.  In  general,  classically-designed  experiments 
respond  to  this  situation  only  by  increasing  the  experimental  error. 

The  economic  aspect  of  experimentation  cannot  be  emphasized 
too  strongly.  Inductive  inference  from  experimental  data  is  subject 
t:o  error  that  can  be  quantified  with  mathematical  statistics;  therefore, 
a measure  of  accuracy  is  obtainable.  In  practice,  it  is  necessary  to 
consider  the  cost  of  obtaining  a particular  accuracy  and  at  what  stage 
the  cost  of  obtaining  increased  accuracy  is  too  great. 

Energy  Exploitation 

Materiel  items  that  are  used  for  intrusion  detection,  target-position 
location,  and  target  discrimination  must  contain  sensors  that  function 
by  exploiting  a wide  range  of  energy  propagation  forms,  such  as  seismic, 
acoustic,  magnetic,  electromagnetic,  etc.  In  general,  an  item  is 
designed  to  exploit  the  energy  generated  by  a target  of  military  interest. 
To  be  exploited,  the  generated  energy  must  be  propagated  from  the  target 
to  the  sensor.  The  informational  content  of  the  propagated  energy,  i.e. 
the  information  at  the  sensor,  is  acted  on  by  the  signal  processor  (i.e. 
the  logic)  of  the  materiel  item.  The  generation  of  energy  in  all  the 
various  forms  is  affected  by  interaction  of  the  generator  (target)  and 
the  environment,  and  the  energy  wave  form  is  further  affected  by  the 
medium  through  which  it  propagates.  Every  large  region  of  the  world 
exhibits  a wide  variety  of  terrain  conditions,  each  of  which  may  change 
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the  character  of  the  wave  in  one  or  more  ways.  If  the  wave  is  changed  to 
a form  which  the  sensor  logic  has  not  been  designed  to  "recognize, " the 
sensor  will  not  respond. 

The  implication  of  the  above  is  that  to  design  an  adequate 
experiment  to  evaluate  the  item’s  performance,  its  interactions  with 
the  operational  environment  must  be  understood  quantitatively.  Prerequisite 
to  this  understanding  is  an  identification  (and  quantification)  of  the 
factors  of  the  environment  that  control  or  significantly  influence  the 
interactions.  Thus,  two  major  experimental  design  steps  emerge: 

(a)  modeling  the  item-terrain  interaction  and  (b)  quantification  of 
the  test  environment. 


Materiel  Item  Design 

The  design  of  advanced  hardware  often  requires  use  of  specialized 
technology,  and  most  designs  emerge  from  defense  contractors,  who  may 
have  proprietary  rights  to  software  and  procedures  they  have  developed. 
Because  of  idiosyncrasies  in  the  design  procedures,  the  contractors 
often  optimize  the  device  for  certain  environments.  Unfortunately, 
testers  may  not  be  aware  of  this  because  they  may  not  have  access  to 
critical  information  on  the  rationale  and  procedures  used  to  develop 
the  item's  design.  Furthermore,  the  as-built  specifications  are  not 
always  provided  to  the  test  agency.  It  is  the  tester’s  duty  to  be 
skeptical,  and  to  try  to  stress  the  item  in  a manner  analogous  to  actual 
use  conditions. 

The  difficulty  in  implementing  this  duty  can  best  be  explained  by 
designing  an  experiment  for  the  evaluation  of  a specific  item.  For 


this  discussion  a seismic  sensor  that  is  required  to  discriminate 
among  classes  of  targets  is  chosen.  In  this  case  it  is  instructional 
to  consider  the  problems  of  designing  sensor  logic  that  can  discriminate 
among  targets. 

The  difficulty  in  designing  such  a seismic  sensor  lies  in  the 
fact  that  sensor  logic  must  be  capable  of  identifying  seismic  signal 
features  that  are  consistently  associated  with  a particular  target, 
regardless  of  terrain  or  target  conditions.  This  generalized  problem 
breaks  down  into  three  components:  (a)  the  inadequacy  of  the  techniques 

available  for  extracting  design  criteria,  (b)  the  inherent  variability 
of  seismic  signals,  and  (c)  the  inadequacy  of  available  seismic  data. 

The  most  popular  method  of  developing  design  criteria  consists  of 
extracting  candidate  signal  characteristics  from  an  empirically  generated 
data  base.  Briefly,  the  steps  are  as  follows: 

a_.  Establish  signature  design  data  base.  This  step  involves 
measuring  signature  data  from  targets  of  interest  operating 
at  various  speeds  and  in  various  terrain  conditions, 
b.  Digitize  the  data  and  separate  them  into  two  batches. 

Select  candidate  discriminating  features  measurable  from 
the  time-  or  frequency-domain  signals.  Examples  include 
ratios  of  energy  in  selected  frequency  bands,  Fourier 
coefficients,  number  of  zero  crossings,  peak-to-peak 
ratios,  root  mean  square  values  of  selected  frequency 
ranges,  mean  values  of  selected  frequency  ranges,  etc. 
d.  Measure  candidate  features  from  one  batch  of  the  signature 
data. 
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Correlate  features  with  target  classes.  Multiple  corre- 
lation techniques  are  used  in  this  step  to  identify  the  most 
persistent  features  and  relate  them  to  target  classes. 
f_.  Test  the  correlation  derived  in  step  e^.  This  step 

involves  testing  the  model  against  the  second  batch  of 
signature  data  not  used  in  the  development  of  the  model. 

The  results  are  often  shown  as  a probability  of  classi- 
fication by  target  class. 

Because  these  techniques  must  rely  on  a finite  number  of  samples, 
they  do  not  ensure  that  the  logic  will  function  for  all  situations  in 
the  total  signature  population,  i.e.  the  data  base  may  not  have  statistical 
representativeness.  As  a point  of  fact,  a statistically  valid  data  base 
may  be  exceedingly  difficult  to  define.  If  the  design  data  base  is 
inadequate,  the  adequacy  of  resulting  sensor  design  is  subject  to 
question. 

Evaluation  Test  Design 
Variables  to  be  considered 

Because  the  design  technique  is  subject  to  statistical  probability, 
the  problem  facing  the  test  designer  becomes  one  of  developing  a scheme  to 
establish  the  performance  envelope  of  the  sensor,  i.e.  demonstrate  under 
what  operational  and  environmental  conditions  the  sensor  will  and  will 
not  meet  the  design  specifications  as  stated  in  the  ROC.  Conventional 
empirical  test  procedures  in  themselves  are  wholly  inadequate  to  define 
the  performance  envelope  of  a sensor  capable  of  discriminating  among 
targets.  For  example,  consider  a sensor  designed  to  discriminate  among: 
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Tracked  vehicles 


cl  • 

b.  Wheeled  vehicles  * 

c_.  Man  or  men 

cL  Rotary  aircraft 

e_.  Fixed-wing  aircraft 

f Noise 

To  decide  the  number  of  test  interactions  required  to  positively  define 
the  sensor's  performance  envelope,  the  sources  of  variance  in  the  signal 
characteristics  (which  are  correlated  with  target  class)  must  be  examined. 
Within  a given  target  class,  such  as  tracked  vehicles,  a number  of  types 
exist,  e.g.  the  M113  personnel  carrier,  the  M60  tank,  etc.  Since  the 
various  types  within  a class  do  not  have  identical  engines,  drive  trains, 
suspensions,  etc.,  it  is  very  reasonable  to  assume  that  values  for  specific 
signal  characteristics  (i.e.  within  a given  terrain  and  at  a given  distance 
from  the  source)  will  not  necessarily  be  the  same  for  each  type  of  tracked 
vehicle.  Thus,  some  variance  in  the  values  of  signal  characteristics 
occurs  because  all  tracked  vehicles  are  not  the  same. 

A similar,  but  more  subtle,  variation  in  the  values  of  specific 
signal  characteristics  may  occur  because  all  members  of  a particular  type 
of  target  within  a class  may  not  have  exactly  identical  characteristics. 

For  example,  all  M113  tracked  personnel  carriers  certainly  have  similar 
power  supplies,  drive  trains,  suspensions,  etc.;  however,  their  physical 
characteristics  (such  as  spring  constants,  effective  horse  power,  weight, 
etc.)  may  not  be  identical  because  of  inherent  variability  in  the  manu- 
facturing of  the  vehicle  and  the  different  degrees  of  wear  and  histories 
of  usage. 
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A third  factor,  and  perhaps  one  of  the  more  important,  is  the 

0r 

influence  of  terrain  conditions  on  the  values  for  the  signal  characteristics 
for  a given  member  of  a specific  target  type  and  within  a specific 
target  class.  Three  general  phenomena  must  be  considered: 

<a.  Generation  of  the  signal. 

b^.  Influence  of  surface  and  subsurface  conditions  on  signal 
characteristics . 

£.  Influence  of  surface  features  (topography)  on  signal 
characteristics . 

It  must  be  emphasized  that  the  influence  of  surface  and  subsurface 
conditions  on  signal  characteristics  is  a function  of  distance  from  the 
source  in  many  cases.  Thus,  the  influence  of  the  terrain  can  be  complex 
indeed. 

The  generation  of  seismic  signals  is  a complex  process  and 
depends  primarily  on  the  target  and  the  terrain  conditions  on  which  the 
target  superimposes  an  input  stress.  When  the  target  is  moving,  it  is 
reacting  to  the  irregularities  in  the  surface  of  the  terrain,  and  as  a 
function  of  time  is  passing  onto  and  over  a variety  of  the  surface 
irregularities.  The  target,  such  as  a vehicle,  will  react  in  various 
ways  to  different  sizes  and  shapes  of  surface  irregularities  and 
therefore  will  produce  variations  in  the  generated  signals  as  a function 
of  time.  In  addition  to  this  time/geometry  problem,  the  signal  generation 
process  is  affected  by  the  subsurface  terrain  conditions;  the  coupling  of 
energy  into  the  terrain  materials  by  the  target  is  not  the  same  for  all 
subsurface  terrain  conditions.  Thus,  the  characteristics  of  the  generated 
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signals  can  vary  because  of  the  variations  in  the  energy  coupling  phenomena 
in  different  subsurface  materials  (both  configurations  and  properties). 

The  effects  of  both  the  surface  irregularities  and  the  subsurface 
characteristics  are  complicated  by  another  variable,  the  speed  of 
travel  of  the  target.  Thus,  the  variation  in  the  generated  signal 
due  to  surface  and  subsurface  effects  may  have  an  additional  component 
of  variation  with  changes  in  target  speed. 

In  addition  to  a consideration  of  signal  generation,  it  is  necessary 
to  examine  signal  propagation.  Once  energy  is  coupled  to  the  medium, 
it  propagates  away  from  the  source  in  various  modes.  As  the  signals 
propagate,  the  terrain  materials  through  which  they  travel  alter  the 
frequency  and  amplitude  characteristics  of  the  signals  by  acting  as  a 
filter.  The  filtering  effect  of  the  terrain  materials  is  a function  of 
distance,  thus  variance  in  measured  signal  characteristics  can  occur  (for 
a given  target)  because  of  different  terrain  conditions  and  as  a function 
of  the  distance  from  the  source  at  which  the  signals  are  measured. 
Furthermore,  surface  irregularities  come  into  play  again,  i.e.  the 
signals  propagating  near  the  surface  may  be  altered  by  reflection, 
refraction,  and  conversion  from  one  propagation  mode  to  another  as  a 
result  of  the  interaction  of  the  signal  and  the  surface  irregularities. 

An  additional  source  of  variation  occurs  if  terrain  conditions  change 
between  the  source  and  the  point  of  measurement.  Thus,  variations  in 
terrain  conditions  in  general  are  the  cause  of  many  sources  of  variation 
in  measured  signal  features.  Another  complication  can  be  added  by  noting 
that  many  terrain  parameters,  such  as  soil  moisture  content  and  soil 
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strength,  may  vary  considerably  (i.e,  at  one  position)  because  of  changes 
in  seasonal  or  climatic  phenomena  (e,g.  rainfall,  freezing,  etc.). 

Other  sources  of  variance  in  signal  characteristics  exist,  such  as 
testing  or  measurement  errors;  however,  elaboration  on  these  topics  is 
beyond  the  scope  of  this  discussion.  The  cogent  question  to  be  answered 
is:  How  can  these  sources  of  variance  in  signal  characteristics  (upon 

which  the  sensor  design  is  predicated)  be  isolated  and  their  effects  be 
accounted  for  in  a test  program  to  evaluate  the  performance  of  the  sensor? 
Empirical  evaluation 

If  the  assumption  is  made  that  the  evaluation  can  be  made  by 
empirical  testing,  it  is  pertinent  to  examine  the  influence  of  the  many 
sources  of  variance  on  the  number  of  tests  required.  In  any  empirical 
study  it  is  necessary  to  collect  sufficient  data  to  define  the  variation 
of  a given  variable  under  a specified  set  of  conditions.  For  example, 
the  seismic  response  (i.e.  the  amplitudes  and  frequencies  of  the  seismic 
signal)  of  a specific  M113  tracked  vehicle  has  some  distribution  for  a 
given  set  of  terrain  and  test  conditions.  Since  the  initial  estimates 
of  variance  values  are  not  readily  available,  statistical  theory  cannot 
be  used  to  calculate  the  number  of  tests  necessary  to  achieve  an  adequate 
evaluation.  For  this  reason,  a somewhat  cursory  analysis  must  be  made 
by  listing  the  relevant  variables  and  estimating  the  number  of  combinations 
of  variables  that  must  be  tested. 

As  stated  earlier,  both  surface  and  subsurface  terrain  conditions 
affect  the  generation  and  propagation  of  seismic  energy.  Further, 
these  conditions  (i.e.  surface  and  subsurface)  are  dynamic  phenomena  that 
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are  closely  related  to  soil  moisture  content.  Also,  the  seismic  signals 
are  affected  by  the  propagating  medium  as  a function  of  range.  Let  it 
be  assumed  that  the  entire  spectrum  of  terrain  surface  conditions  of 
interest  can  be  represented  by  10  specific  situations,  and  the  spectrum  of 
subsurface  terrain  conditions  of  interest  can  be  limited  to  100  specific 
conditions.  Since  both  surface  and  subsurface  conditions  are  dynamic 
phenomena  that  are  closely  related  to  moisture  content,  let  us  assume 
that  five  different  situations  (e.g.  five  moisture  conditions,  etc.) 
can  occur.  Also,  let  us  assume  we  need  measurements  at  10  distances 
from  the  target  to  the  source.  In  the  extreme,  but  considering  only 
these,  factors,  the  number  of  possible  combinations  that  must  be  tested 
is  a striking  50,000. 

If  this  were  not  bad  enough,  consider  the  fact  that  it  is 
necessary  to  define  the  variability  in  the  signal  characteristics 
that  may  occur  for  an  individual  target  of  a specific  type  (e.g.  a 
specific  M113) . To  do  this  let  us  assume  the  need  for  five  repeat 
trials.  Also,  since  all  individuals  may  not  react  the  same,  let  us 
test  five  individuals  of  each  target  type  within  each  target  class. 

In  addition  to  this,  it  must  be  remembered  that  there  are  numerous 
types  within  each  class,  say  five.  Finally,  we  are  dealing  with  six 
classes  of  targets.  When  all  of  these  combinations  are  considered, 
the  resultant  number  of  combinations  (or  required  field  tests)  could 
total  37,500,000.  Clearly,  this  is  not  a viable  solution,  either 
technically  or  economically,  and  an  alternative  approach  must  be  sought. 
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Empirical-theoretical  evaluation 

Perhaps  the  most  viable  solution  consists  of  a balanced  experimental 
and  theoretical  program.  In  this  approach,  well-controlled  empirical 
tests  are  conducted  to  ensure  that  the  hardware  functions,  i.e.  it  meets 
ruggedness  and  longevity  specifications  and  the  electrical  circuits  work. 
Equally  important,  the  empirical  tests  demonstrate  how  the  device  works 
in  a specific  set  of  test  conditions. 

In  the  theoretical  portion  of  the  program,  realistic  simulation 
models  are  used  to  estimate  how  the  device  would  function  if  the  various 
terrain  and  target  factors  were  varied  throughout  the  range  of  interest. 
The  deficiency  in  applying  the  balanced  approach  centers  around  the  fact 
that  simulation  models  adequately  describing  sensor  performance  as  a 
function  of  target  and  environmental  conditions  are  not  readily  available. 
Further,  for  practical  applications,  they  must  be  formulated  such  that 
they  accept  unique  and  measurable  target  and  environmental  factors. 
Although  difficult,  formulation  of  adequate  simulation  models  is  both 
possible  and  practical.  To  illustrate,  the  following  paragraphs  briefly 
describe  a seismic  sensor  performance  model  developed  at  the  WES.  Also 
presented  are  examples  of  how  well  the  signals  predicted  with  the  model 
compare  with  measured  signals  from  man-walking  and  vehicle  targets. 

Also,  presented  are  examples  of  how  the  signals  change  as  a function 
of  terrain  conditions. 
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WES  Seismic  Sensor  Performance  Model 


The  simulation  process 

The  seismic  prediction  chain  for  the  simulation  process  developed 
at  the  WES  is  shown  in  fig.  1.  Stress  signals  are  predicted  by  the 
various  intruder  models  (e.g.  footstep  and  wheeled  and  tracked  vehicles) 
for  the  forces  applied  to  the  ground  media  as  the  intruder  travels  over 
it.  The  stress  signals  are  used  by  the  microseismic  signal  model  to 
compute  microseismic  signals.  The  microseismic  signals  are  applied  to 
the  seismic  sensor  model*  which  is  used  to  compute  sensor  response  as  a 
function  of  site  and  target  properties.  The  simulation  models  are 
described  in  detail  in  the  WES  report  entitled  "Effects  of  Environment 
on  Microseismic  Wave  Propagation  Characteristics  in  Support  of  SID 
Testing  at  Fort  Bragg,  N.  C.;  Report  2,  Comparison  of  Summer-  and  Winter- 
Season  Conditions,"  by  T.  L.  Engdahl  and  H.  W.  West,  soon  to  be  published. 

It  should  be  noted  that  the  model  simulating  the  sensor  can  be  in 
the  form  of  a mathematical  transfer  function,  or  the  predicted  signal 
can  be  input  via  magnetic  tape  directly  into  the  sensor  itself.  For 
this  reason  the  sensor  does  not  necessarily  have  to  be  modeled,  and  the 
critical  link  in  the  simulation  process  is  the  prediction  of  the  seismic 
signal.  Fig.  2 demonstrates  how  well  this  can  be  accomplished  for  a 
sLgnal  resulting  from  a footstep.  The  two  sets  of  measured  curves 
(figs.  2a  and  2b)  are  for  the  same  walking  man  at  the  same  Fort  Bragg, 

N.  C.,  site,  but  the  second  set  (fig.  2b)  was  recorded  after  a heavy  rain 
some  10  days  later  than  the  first  (fig*  2a).  In  the  soils  at  Fort  Bragg 
(predominantly  sand),  the  footstep  signals  have  approximately  the  same 
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amplitude  and  frequency  content  in  both  the  wet  and  the  dry  conditions. 

This  is  not  necessarily  the  case  for  all  soils,  but,  under  this  situation 
an  intermediate  condition  ought  to  show  roughly  the  same  signal  character- 
istics. The  predicted  signals  (fig.  2c)  are  for  such  an  intermediate 
condition;  the  surface  rigidity  data  for  the  footstep  model  were  collected 
a short  time  after  both  sets  of  footstep  signal  measurements  were  taken, 
when  the  soil  moisture  content  was  intermediate  to  before-rain  and  after- 
rain conditions.  It  can  be  seen  that  the  predicted  signals  have  wave 
forms  with  characteristics  (amplitude,  frequency  content,  and  signal 
duration)  similar  to  those  in  both  sets  of  measured  signals. 

Measured  and  predicted  signals  from  Fort  Bragg  for  an  M151  jeep 
at  32  km/hr  are  compared  in  figs.  3 and  4.  The  measured  and  predicted 
time-domain  signals  in  fig.  3 show  very  good  agreement  at  ranges  of  50, 

100,  150,  and  200  m.  For  this  particular  set  of  predictions,  the  signals 
are  primarily  generated  by  the  suspension  as  the  jeep  travels  cross-country. 
If  the  vehicle  had  been  traveling  much  slower  or  traveling  over  a smooth 
surface,  the  suspension  component  would  be  reduced  and  the  seismic  signal 
would  reveal  the  engine  signal  components.  Both  the  predominant  frequency 
and  the  amplitudes  are  reduced  as  the  range  increases.  This  is  shown 
more  clearly  in  the  frequency-domain  signals  for  the  same  test  (fig.  4). 

As  the  range  increases,  the  high-frequency  signal  components  are  reduced 
in  amplitude  at  a much  greater  rate  than  the  low-frequency  components. 

This  causes  the  peak  in  the  spectrum  to  reduce  in  amplitude  and  shift 
to  lower  frequencies. 

The  terrain  inputs  required  for  the  man-walking  target  predictions 

compression  spring  constants  (k  ) and  deflection  at  maximum 

S j C 


are : 


bearing  capacity  (zmax)  obtained  from  plate-load  tests;  and  thickness  (T) , 

compression  wave  velocity  (Vp),  shear  wave  velocity  (Vg)>  of  the  various 

soil  layers  as  defined  by  seismic  refraction  surveys,  and  soil  wet 

density  (p).  Normally  good  comparison  of  predicted  and  measured  signals 

can  be  obtained  if  only  the  first  and  second  layers  are  considered.  In 

addition  to  the  surface  parameters  (i.e.  k and  Z ) discussed  above 

s,c  max 

for  a man-walking  target,  a surface  geometry  profile  is  required  as  an 
input  to  both  the  wheeled  and  tracked  vehicle  models.  Subsurface  data 
(i.e.  T,  V , V ),  and  p)  requirements  are  identical  to  that  required  for 

r o 

making  a prediction  for  the  signal  from  footsteps. 

Extrapolation  of  test  results 

Extrapolation  of  test  results  to  environmental  conditions  outside 
the  test  area  is  accomplished  by  varying  the  environmental  factors 
discussed  in  the  preceding  paragraph.  For  example,  various  combinations 
of  the  factors  representing  soft  soil,  firm  soil,  and  frozen  ground  are 
shown  in  figs.  5,  6,  and  7.  Predicted  time-  and  frequency-domain  signals 
for  a man-walking  target  at  a range  of  5 ra  is  shown  in  fig.  8.  These 
examples  (which  cover  a wide  range  of  soil  conditions)  show  that  the 
particle  velocity  amplitude  for  the  frozen  ground  is  about  two  orders 

_3 

of  magnitude  (from  ~ 20  to  .2  cm/sec  x 10  ) less  than  that  for  the  soft 
soil.  Also,  the  energy  is  propagated  at  higher  frequencies  as  the  soil 
rigidity  increases. 

Fig.  9 shows  predicted  results  for  a Soviet  light  tank  (PT76) 
on  the  same  terrain  conditions  as  in  the  man-walking  predictions. 

The  speed  of  the  vehicle  is  5 mph,  and  it  is  at  a range  of  75  m from 
the.  sensor.  These  plots  show  dramatically  how  the  shape  of  wave  forms 
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from  a vehicle  depends  on  terrain  conditions.  As  with  footstep  signals, 
the  amplitude  decreases  with  soil  rigidity.  Also,  the  dominant  frequency 
increases  with  soil  rigidity. 

Conclusions 

It  must  be  recognized  that  the  various  terrain  and  target  parameters 
can  combine  so  as  to  have  a synergistic  effect  on  the  resultant  wave 
forms;  therefore,  many  combinations  of  terrain  factors  must  be  evaluated. 
The  WES  sensor  performance  models  use  algorithms  that  can  be  solved 
efficiently;  therefore,  they  provide  a means  for  generating  a relatively 
large  number  of  predictions  at  a low  unit  cost.  Work  is  now  being  directed 
toward  devising  a listing  or  matrix  of  terrain  factors  to  provide  a data 
base  for  the  comprehensive  evaluation  of  any  seismic  sensor.  It  is  felt 
that  this  balanced  approach,  i.e.  balance  between  empirical  testing  and 
theoretical  extrapolation,  can  be  directly  applied  to  the  evaluation  of 
many  items  of  advanced  materiel. 
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FIGURE  1 
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SHORT  PULSE  TESTING  OF  EEDs 
MD 

THE  BRUCETON  PROBLEM 


Ramie  H.  Thompson 

The  Franklin  Institute  Research  Laboratories 

and 

Burton  V.  Frank 

Nuclear  Engineering  Directorate 
Picatinny  Arsenal 

1 . INTRODUCTION 

The  authors  have  recently  completed  tests  of  the  short  pulse  response 
of  some  electroexplosive  devices.  The  primary  results  were  evaluated 
using  the  classical  electroexplosive  device  statistical  techniques.  This 
paper  describes  the  test  equipment  and  technique  in  some  detail.  Hopefully, 
enough  detail  to  allow  the  reader  to  get  a feeling  for  the  accuracy  of 
the  techniques  and  yet  not  bore  him  completely.  In  any  case  the  section 
describing  the  stimuli  and  their  generation  can  be  skipped  without  any 
great  loss  of  knowledge  about  the  central  question  we  wish  to  raise.  That 
question  is  straightforward.  Are  the  commonly  used  statistical  techniques 
of  electroexplosive  evaluation  adequate  for  the  use  to  which  we  put  the 
information  generated  by  these  techniques? 

The  authors  suspect  that  the  presently  used  techniques  are  the  best 
of  a bad  lot-the  result  of  economic  and  theoretical  compromiae-and  frankly 
seek  suggestions  for  improvement  or  alternate  techniques. 

2 . BACKGROUND 

The  Applied  Physics  Laboratory  of  The  Franklin  Institute  Research 
Laboratories  has  been  involved  with  the  evaluation  of  the  DC  and  RF  (both 
pulsed  and  continuous  wave)  responses  of  Electroexplosive  Devices  (EED’s) 
for  about  twenty  years.  Recently  an  interest  in  the  response  of  EED’s  to 
very  short  duration/high  amplitude  electrical  stimuli  has  been  prompted  by 
concern  about  possible  Electromagnetic  Pulse  (EMP)  interactions  with  EED’s. 
The  original  work  we  performed  in  this  field  used  damped  sinusoidal  stimuli 
but  Interest  has  shifted  to  the  more  easily  produced  and  controlled  rec- 
tangular pulse  shape.  All  of  the  work  discussed  here  uses  the  short  rec- 
tangular pulse  as  the  basic  EED  stimuli. 
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How  short  is  a short  pulse?  We  have  conducted  extensive  tests  using 
25  ns,  50  ns  and  100  ns  pulses  but  pulse  lengths  can  be  increased  without 
trouble  to  about  10  microseconds.  The  primary  advantage  of  our  specialized 
pulse  supplying  equipment  is  the  ability  to  monitor  our  high  amplitude 

J, 

stimuli  and  responses  without  interference. 

3.  THE  STIMULI  AND  THEIR  GENERATION4- 

Conventional  type  twin  lead  EED's  can  be  initiated  by  two  ways: 

(1)  Passing  current  through  the  bridgewire  (the  conventional  firing 
mode)  and 

(2)  Application  of  a high  voltage  between  the  pins  of  the  EED  and 
the  metal  case. 

Our  pulse  generating  equipment  can  supply  short  pulses  to  the  EED  in 
either  of  these  firing  modes.  Figure  1 shows  a typical  high  current  short 
duration  pulse  applied  to  the  bridgewire  of  a conventional  type  hot  wire 
EED.  The  oscilloscope  traces  shown  here  have  been  traced  from  the  actual 
oscilloscope  photographs.  Note  that  the  waveform  on  the  left  shows  the 
current  thourgh  the  bridgewire  of  the  EED  and  the  waveform  on  the  right 
shows  the  voltage  across  the  bridgewire.  The  time  scale  is  the  same  on 
both  photographs  and  the  individual  sweeps  start  at  the  same  time. 

Figure  2 shows  a high  voltage  short  pulse  applied  to  the  pin-to-case 
firing  mode  of  an  EED.  Note  that  the  current  is  very  low  and  that  this 
is  associated  with  a ’’no-fire"  response  of  the  EED.  This  current  is  that 
which  actually  flows  between  the  pins  and  the  case  of  the  EED  during  the 
voltage  pulse  application. 

Figures  3 and  4 show  simplified  equivalent  circuit  schematics  for  the 
pin-to-pin  and  pins-to-case  short  pulse  testing  equipment  configurations 
and  Figure  5 is  an  overall  schematic  for  the  actual  system. 


^The  short  pulse  generating  equipment  is  described  in  great  detail  in 
"Pins -to -Case  Short  Pulse  Sensitivity  Studies  for  the  Atlas  PC  Switch," 
FIKL  report  I-C3410  produced  for  Picatinny  Arsenal. 

Most  of  the  material  in  this  section  is  taken  directly  from  "Short  Pulse 
Testing  of  EED's,"  a paper  by  R.  H.  Thompson  given  at  the  8th  Symposium 
on  Explosives  & Pyrotechnics  at  The  Franklin  Institute,  Phila.,  Pa.  in 
Feb.  1974. 
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Figure  3.  Equivalent  Circuit  for  Pin-to-Pin  Exposures 


112 


Figure  4.  Equivalent  Circuit  for  Pin-to-Case  Exposures 
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Figure  5.  The  Overall  System  Schematic 


In  operation  the  equipment  is  used  in  this  order:  (Refer  to 

Figure  5) . 

1)  Adjust  the  high  voltage  supply  for  the  desired  voltage  and 
polarity.  This  charges  transmission  lines  1 and  2 through 
the  large  resistors  R^  and  R2. 

2)  The  closure  of  pushbutton  P energizes  time  delay  relay  R2  for 
approximately  2 seconds. 

3)  Relay  K2  energizes  the  40  KV  vacuum  relay  K1  for  2 seconds. 

4)  As  the  contacts  of  begin  to  close  they  eventually  reach  a 
point  where  the  charged  transmission  line  (line  2)  arcs  to  the 
movable  contact  of  Kl.  This  arc  effectively  connects  trans- 
mission lines  2 and  3 together.  This  switching  action  is 
roughly  equivalent  to  that  of  a perfect  switch  so  we  approximate 
the  theoretical  situation  of  a charged  transmission  line  connected 
instantaneously  to  a load. 

5)  The  electromagnetic  disturbance  caused  by  the  switching  propagates 

from  the  relay  contacts  in  two  directions:  both  through  line  3 

toward  the  insertion  unit  and  back  through  line  2 toward  R^ . 

R^  is  chosen  to  be  very  large  in  relation  to  the  50  ohm  charac- 
teristic impedance  of  line  2 so  that  it  approximates  an  open 
circuit.  The  insertion  unit  contains  a series  or  parallel 
resistor  such  that  the  impedance  looking  toward  the  EED  is 
about  50  ohms.  For  pin-to-pin  tests  the  insertion  unit  con- 
tains a 50  ohm  resistor  in  series.  For  pin-to-case  tests  a 
55  ohm  parallel  resistor  is  usually  used.  In  general  the  choice 
of  impedance  insures  that  the  system  behaves  as  if  line  2 is 
open  circuited  at  the  R^  end  and  loaded  with  50  ohms  at  the 
relay  end . This  choice  results  in  a theoretically  rectangular 
voltage  across  the  input  impedance  to  the  insertion  unit.  The 
aptitude  of  the  pulse  if  theoretically  one-half  the  DC  charging 
voltage  and  its  duration  is  twice  the  one  way  delay  time  of 
line  2 . 

6)  The  damped  rectangular  stimulus  is  coupled  through  a teflon  filled 
coaxial  cable,  through  the  T&M  monitor,  through  the  teflon  filled 
coaxial  firing  mount  and  thus  to  the  EED. 

7)  The  voltage  and  current  for  the  EED  are  monitored  by  the  T&M 
monitor  and  coupled  to  the  oscilloscope  by  fairly  short  coaxial 
cables.  The  traces  of  the  scopes  are  photographed  on  high  speed 
film. 


Note  that  the  block  diagram  shows  several  components  in  an  oil  bath. 
The  oil  used  is  standard  transformer  oil.  It  eliminates  corona  discharge, 
and  Its  resulting  interference,  from  the  system.  The  metal  cans  shown  in 
the  block  diagram  are  constructed  to  completely  confine  the  electromagnetic 
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noise  due  to  the  arc  at  the  switch.  Note  that  the  only  connections  between 
the  shielded  volume  containing  the  arc  and  the  outside  volume  containing  the 
oscilloscopes  is  through  the  300  feet  of  RG  14/11,  thourgh  the  300  feet  of 
shielded  Twinax  or  though  the  voltage  and  current  monitor  cables.  Noise  on 
the  monitor  cables  is  a real  perturbation  to  our  measured  signals  and  it  will 
be  displayed  on  the  scope  as  it  should  be.  The  other  type  of  noise  that  would 
normally  affect  our  oscilloscopes,  due  to  direct  coupling  between  the  arc 
and  the  scopes,  does  not  show  on  our  photographs  because  it  must,  in  our 
system,  propagate  at  least  the  length  of  the  RG  14/11  and/or  the  Twinax 
before  it  can  interact  with  the  scope.  This  takes  enough  time  that  our 
photography  is  finished  and  the  scopes  have  finished  their  sweep  long  before 
the  noise  arrives.  We  thus  get  clean  pictures  of  the  actual  voltage  and 
current  across  the  load. 

The  two  metal  shielding  cans  shown  in  Figure  5 differ  in  construction. 

The  can  that  contains  R^  is  a modified  one  gallon  paint  can.  The  shield 
line  1 is  soldered  360  degrees  to  the  bottom  of  the  can  which  it  penetrates. 
The  lid  of  the  can  is  penetrated  by  line  2 and  this  shield  is  also  soldered 
360  degrees.  In  use  R£  is  soldered  in  place  with  the  lid  almost  closed,  the 
can  is  filled  with  oil,  the  lid  closed  and  the  can  submerged  in  an  oil  bath. 
The  can  containing  relay  K1  is  a modified  50  caliber  ammunition  can.  The 
can  has  both  input  and  output  connectors.  They  are  Teflon  insulated,  General 
Radio,  50  ohm  connectors.  RF  gasketing  material  has  been  applied  to  the 
lid/body  mating  line.  The  overcentered  closure  device  thus  makes  a good  RF 
joint  when  the  can  is  closed.  The  shield  of  the  twin  lead  relay  control 
cable  is  360  degree  soldered  where  it  penetrates  the  can.  Line  2 is  dressed 
with  a Teflon  insulated  connector  on  the  end  opposite  the  paint  can  lid. 

This  connector  mates  with  the  input  connector  on  the  ammo  can.  In  use  the 
can  with  its  input  connector  is  submerged  in  an  oil  bath.  The  output  con- 
nector is  above  the  oil  level. 

Note  that  line  2 is  constructed  with  a paint  can  lid  soldered  at  one 
end  and  a connector  at  the  other  end.  The  length  of  this  line  controls 
the  pulse  length  of  the  overall  system.  We  have  constructed  lines  giving 
25 , 50  and  100  nanosecond  pulses  and  with  the  arrangement  described  above 
we  can  substitute  one  for  another  in  about  five  minutes. 
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The  T&M  monitor  contains  a shunt  506  ohm  voltage  divider  and  a 0.051  ohm 
series  resistor  for  current  monitoring.  The  voltage  division  ratio  is  98.6:1 
and  the  output  impedance  is  50  ohms . The  voltage  output  of  the  T&M  monitor 
is  coupled  through  six  feet  of  50  ohm  line  to  23  dB  of  General  Radio  50  ohm 
pads.  These  pads  are  in  turn  connected  through  five  feet  of  50  ohm  cable 
to  a shunt  50  ohm  load  at  the  input  to  the  voltage  monitoring  454  Tektronix 
oscilloscope.  The  current  monitoring  output  of  the  T&M  monitor  (which  is 
across  the  0,051  series  resistor)  is  connected  by  24  feet  of  50  ohm  cable 
to  the  input  of  the  current  monitoring  scope.  This  input  is  also  shunted 
by  a 50  ohm  load.  The  current  monitoring  scope  is  externally  triggered 
by  the  voltage  monitoring  scope’s  sweep  gate  signal  through  a six  foot  50  ohm 
cable.  The  voltage  monitoring  scope  is  internally  triggered  by  the  voltage 
input.  The  cable  lengths  are  critical  (within  a few  feet)  for  proper  time 
relation  of  the  voltage  and  current. 


The  98,6:1  voltage  divider  and  the  23  dB  pad  result  in  an  equivalent 
deflection  factor  for  the  voltage  monitoring  scope  of: 


Stimulus 

Volts/Division 


Scope 

Volts/Division 


1393 

2786 

6964 


1 

2 

5 


The  0.051  current  monitoring  series  resistor  results  in  an  equivalent 
deflection  factor  for  the  current  monitoring  scope  of 


Stimulus 
Amps/Di  vision 

0.98 

1.96 

3.92 

9.8 

19.6 

39.2 

98.0 

196.0 


Scope 

Volts/Division 

0.05 

0.1 

0.2 

0.5 

1.0 

2.0 

5.0 

10.0 
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Since  the  reading  of  the  oscilloscope  pictures  can  seldom  be  done  to 
better  than  5%  accuracy  the  calibration  factors  above  are  rounded  oft  In 
the  quotation  of  scope  deflection  factors.  Thus  we  give  40  amperes  per 
division  for  an  actual  39.2  ampere  per  division  factor  and  we  quote  2800 
volts  per  division  fot  an  actual  factor  of  2786  volts  per  division,  etc. 

The  scopes  are  calibrated  against  their  internal  voltage  standards 
and  the  system  monitoring  equipment  is  calibrated  (or  checked)  by  connecting 
a 50  ohm  load  on  the  end  of  the  T&M  monitor  and  observing  the  magnitude 
of  voltage  and  current  indicated.  During  this  check  the  horizontal  posi- 
tioning controls  of  the  scopes  are  adjusted  so  the  pulses  begin  at  the  same 
place  on  the  scope  faces.  This  facilitates  comparison  of  the  voltage  and 
current  photographs. 

Considerable  short  rectangular  pulse  testing  has  been  done  on  the  Atlas 
Squib  Switch.  Figure  6 shows  a cutaway  view  of  the  overall  switch  and 
Figure  7 shows  a disassembled  switch.  Most  of  our  testing  was  done  on  the 
"ping”  alone.  This  subassembly  is  indicated  in  Figure  6. 

4.  SOME  RESULTS 

Figure  8 is  a representative  set  of  traces  of  a 25  nanosecond  pin-to-pin 
Bruceton  test  for  the  Atlas  Squib  Switch  plug.  If  we  assume  that  the  bridge- 
wire  resistance  is  constant  during  the  test  (and  it  seems  likely  from  study 
of  the  voltage  trace  In  Figure  8)  we  can  calculate  firing  energies.  Figure  9 
compares  the  mean  firing  energies  for  various  other  Brucetons  with  that  de- 
termined by  our  25  ns  test.  Note  that  the  means  all  roughly  compare.  This 
result  points  to  the  fact  that  the  pin-to-pin  response  to  short  pulses  is 
in  keeping  with  the  theory  applicable  to  longer  pulses  and  that  no  new  phe- 
nomena are  evidenced  in  this  shorter  time  regime. 

Figures  10  and  11  show  double  exposure  results  of  high  voltage  pin-to-case 
short  pulse  tests  on  the  Atlas  Squib  Switch.  Both  photos  show  a normal  "no- 
fire"  response  with  very  small  currents  and  a "fire"  response  that  clearly 
shows  breakdown  and  a large  pin-to-case  current*  A considerable  number  of 
such  photos  shows  that  pin-to-case  initiation  is  a phenomena  that  takes 
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Figure  6.  Switch,  Squib  Actuated.  Non  Delay 
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Figure  9.  Energies  Computed  Using  a Resistance  of  1 .83  Ohms 
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place  on  a nanosecond  time  scale.  Several  determinations  of  the  energy 
delivered  to  the  breakdown  impedance  have  been  made.  The  minimum  energy 
determined  to  date  is  about  25,000  ergs  but  more  work  is  obviously  neces- 
sary to  determine  the  minimum  energy  necessary  for  initiation.  We  feel 
that  continued  work  in  this  area  could  lead  to  a much  better  understanding 
of  overall  pin-to-case  firing  characteristics  of  electroexplosives. 

5.  THE  BRUCETON  AND  THE  PROBLEM 

Figure  12  shows  the  computer  output  from  a not  atypical  Bruceton* 
calculation  using  the  short  pulse  data.  Note  that  we  compute  the  Bruceton 
statistics  using  both  the  "fires''  and  the  "no  fires"  and  then  average  the 
results.  The  confidence  levels  are  computed  using  the  average  data.  The 
computer  program  is  written  in  Fortran  IV  and  can  produce  95%  or  90%  con- 
fidence levels  at  choice.  It  can  also  produce  the  same  sort  of  output 
as  shown  in  Figure  12  for  the  1%,  99%  levels  and  the  10%,  90%  levels. 

Other  levels  can  of  course  be  obtained  but  we  would  need  to  change  the 
program  cards  slightly. 

The  most  common  use  of  the  no-fire  (i.e.  0.1%)  probability  levels 
as  determined  by  the  Bruceton  test  procedure  is  as  an  absolute  safety 
level.  For  example  many  radio  frequency/EED  safety  analyses  use  the 
0,1%  power  level  (with  95%  conf.)  of  a radio  frequency  Bruceton  test  as 
the  absolute  maximum  of  power  that  can  be  coupled  to  the  EED  and  have  the 
overall  system  considered  safe.  Other  analyses  add  a safety  factor  by 
dividing  the  no  fire  level  by  two  or  ten.  In  any  event  a sensitivity 
test  that  predicts  a no-fire  level  that  is  consistently  lower  than  the 
actual  tends  to  err  on  the  conservative  or  safe  side. 

Our  end  uses  of  the  Bruceton  results  mentioned  in  this  report  were 
for  the  estimation  of  "no-fire"  levels.  We  have  used  the  0.1%  level  (with 


*The  "Bruceton"  test  we  refer  to  is  a test  of  the  type  described  in  "Statis- 
tical Analysis  for  a New  Procedure  in  Sensitivity  Experiments,"  a report 
submitted  by  the  Statistical  Research  Group,  Princeton  University,  to  the 
National  Defense  Research  Committee  July  1944, 
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95%  confidence)  as  this  no-five  level. 


Our  primary  questions  about  the  application  of  the  Bruceton  in  this 
manner  are  two : 

1)  How  does  the  error  in  the  determination  of  the  test  levels  influence 
the  Bruceton  results? 

2)  Is  the  Bruceton  test  procedure  a useful  tool  in  this  application 
or  are  there  other  more  "optimum'’  techniques? 
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Figure  12.  Pin-To-Pin  Bruceton  Results 
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ABSTRACT.  The  detection  optimization  problem  can  be  reduced  to 
simple  terms,  even  though  the  specific  technology  used  may  be  sophis- 
ticated. Basically,  the  target’s  response  signal  and  its  contrast  to 
background  is  to  be  maximized  and  it  should  provide  as  unique  a sig- 
nature as  possible.  The  first  two  experimental  conditions  optimize 
target  visibility;  the  latter  minimizes  false  signals.  In  other  words, 
the  objective  is  to  optimize  the  measure  of  a detection  system's  per- 
formance - the  detection  efficiency  and  target  specificity. 

This  paper  provides  a generalized  analysis  of  detection  efficiency 
optimization  in  a system  which  measures  a spectral  response  of  a target 
in  a consistent  background.  The  spectral  response  is  assumed  to  be  a 
Gaussian  shaped  enhancement  mixed  within  a uniform  background.  The 
analysis  related  signal  intensity,  signal  to  background  ratio,  back- 
ground determination  error,  efficiency  and  specificity.  Optimization 
of  the  signal  window  width  and  decision  threshold  are  constraining  con- 
ditions. Calculated  results  and  the  relative  sensitivity  of  each  of 
these  parameters  will  be  presented.  This  analysis  provides  the  detection 
system  designer  with  the  information  needed  to  specify  critical  parameters 
or  to  predict  the  performance  of  a given  system. 

1.  INTRODUCTION.  Another  title  for  this  paper  could  be  "A  Pedes- 
trian's Approach  to  Detection  Theory".  Being  neither  a statistician  nor 
familiar  with  the  state-of-the-art  in  detection  theory,  I have  developed 
an  approach  to  detection  optimization  in  a way  which  makes  sense  to  me 
and  would  like  to  share  with  you. 

The  approach  to  target  detection  which  we  are  going  to  consider  is 
intended  to  be  practical  in  nature  and  is  in  some  respects  model  dependent. 
For  clarity's  sake  only  a simple  but  somewhat  generalized  case  will  be 
described.  Many  of  the  constraining  assumptions  may  be  easily  changed  to 
better  approximate  a particular  technique  with  relatively  simple  modifi- 
cations to  the  analysis. 

The  essence  of  target  detection  is  the  accurate  determination  of 
whether  or  not  a target  is  within  the  field  of  view  of  a detection  system 
- often  under  conditions  in  which  the  signal  intensity  is  quite  limited. 
This  entails  tailoring  the  detection  system  so  as  to  maximize  the  visibi- 
lity of  the  target  and  providing  decision  logic  which  maximizes  the  pro- 
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bability  of  correctly  cueing  on  a target  while  minimizing  the  introduc- 
tion of  false  cues.  This  in  turn  assumes  that  the  detection  system  provides 
a binary  response.  A cue  is  given  if  a target-like  signature  is  detected, 
otherwise  no  cue  is  given.  We  assume  that  this  decision  must  be  rendered 
in  a single  pass  over  the  target  and  based  entirely  on  the  response  of 
a single  detector. 

In  actual  detection  scenarios  an  object  which  simulates  a target  may 
be  encountered,  thereby  inducing  a false  signal.  Because  the  description 
of  this  class  of  false  signals  is  closely  lihked  to  the  specific  technique 
used,  it  will  not  be  further  considered  here,  but  rather  assumed  to  be  an 
isolated  problem  to  be  considered  by  the  detector  designer.  Another  source 
of  false  signals  is  spurious  cueing  due  to  statistical  fluctuations  in  the 
detector  response.  We  assume  this  to  be  the  predominant  consideration  in 
optimizing  detection  specificity. 

The  model  we  will  consider  is  that  of  a target  which  emits  either 
spontaneously  or  by  stimulation  a characteristic  spectral  signature, 
which  in  turn  is  measured  in  one  pass  by  the  detector.  The  reconstructed 
spectrum  may  look  something  like  Figure  1.  It  would  be  composed  of  a 
background  and  a signal  whose  centroid  is  at  a known  position  in  the  spec- 
trum. The  target  emission  or  detector  response  is  assumed  to  be  quantized 
or  quantizable,  thus  the  spectral  intensity  on  this  graph  is  defined  as 
the  number  of  counts,  N,  corresponding  to  the  detected  quanta,  per  interval 
in  the  spectral  parameter,  x,  such  as  counts  per  energy,  wavelength,  or 
time  of  arrival.  Quantized  data  is  particularly  convenient  for  computer 
processing,  which  this  analysis  lends  itself  to.  We  will  assume  that  both 
signal  and  background  count  fluctuations  within  an  interval  obey  Gaussian 
statistics  with  one  standard  deviation  corresponding  to  the  square  root  of 
the  number  of  counts  within  the  interval.  Such  Poisson  fluctuations  are 
adequately  approximated  by  Gaussian  statistics  if  the  number  of  counts  is 
large.  In  addition,  the  background  shape  is  assumed  predictable  so  that  the 
background  under  the^ signal  can  be  inferred  by  normalizing  the  counts  in  a 
pure  background  portion  of  the  spectrum.  For  simplicity  the  background  is 
assumed  flat  in  the  region  of  the  signal  so  that  the  signal  shape  is  not 
distorted.  The  signal  is  assumed  to  have  a Gaussian  shape.  This  is  a good 
approximation  in  many  cases,  particularly  if  the  original  signal  width  is 
much  less  than  the  detection  system  resolution,  so  that  it  is  smeared  into 
the  Gaussian  shape. 

ft.  MODEL  SPECIFICATION.  Figure  2 illustrates  some  of  the  basic 
parameters  we  will  need.  The  resolution  of  many  systems  is  defined  in 
terns  of  the  full  width  at  half  maximum.  For  a Gaussian,  one  standard 
deviation  is  slightly  narrower  than  the  half  width  at  half  maximum  and  a 
half-bin  width  parameter,  xg,  remains  to  be  calculated.  This  parameter 
defines  the  upper  and  lower  bounds  of  the  optimal  signal  window,.  The  sig- 
nal which  falls  within  this  window  is  called  S.  This  window  corresponding- 
ly defines  the  amount  of  background  which  is  collected,  B.  S0  and  B0  are 
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internally  calculated  numbers,  since  they  are  a function  of  Xg,a  cal- 
culated parameter.  In  order  to  externally  specify  the  signal  and  back- 
ground we  bhould  select  experimentally  meaningful  definitions.  So 
accordingly  is  defined  as  the  total  signal  counts  in  the  entire  Gaussian, 
and  is  the  background  counts  within  the  FWHM  interval  of  the  signal. 

The  detection  system  is  designed  to  cue  when  the  total  counts  within 
the  window  exceeds  a certain  threshold.  The  threshold  should  be  as  much 
above  the  background  as  possible  for  good  specificity  - to  avoid  cueing 
on  statistical  fluctuations  - and  below  the  total  signal  plus  background 
for  good  detection  efficiency.  We  can  write  this 

T,  = B + a 
1 s 

T2  = S + B - aA  /$  + B 

where  a is  the  number  of  standard  deviations  above  the  background  and 
a is  tne  corresponding  parameter  below  the  signal  plus  background.  Note 
tfiat  the  a's  are  the  number  of  standard  deviations  and  not  the  standard 
deviation  itself. 

The  error  on  the  background  is  more  than  just  the  statistical  error. 
There  is  also  a determination  error,  since  B must  be  inferred.  Thus  the 
complete  expressions  for  a standard  deviation  are 

T.  - B + a + AB2 
1 s ____ i 

T2=S+B-ae»4+B+  AB2 

where  aB  is  the  determination  error.  In  actuality  Ti  = T2  which  is  the 
detection  threshold,  T,  and  it  is  optimized  when  both  a’s  are  maximized. 

The  specific  detection  problem  determines  the  relative  importance  of 
efficiency  and  specificity.  This  ratio  is  designated,  k,  the  signal  t&d> 
background  ratio  is  p,  and  0 is  the  ratio  of  background  measurement  error 
to  statistical  error. 


a 


3 E AB_ 

& 

Making  these  substitutions  and  setting  Tx  = T2  we  obtain  the  following 
expression. 
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1 


- <f>  (p»  s)  = 


J\  + g + Kv\  + p + 


Since  the  a\s  are  to  be  maximized  the  expression  <f>  should  be  minimized. 


The  signal  is  written  as  a function  of  the  spectral  parameter  x 
whose  origin  is  assumed  to  be  the  signal,,  centroid. 

s°  4M 

S (x)  = e T|xoJ 

x0  /Ti 

So  is  the  total  signal  counts  and  as  before  x is  the  Gaussian's  width 
expressed  as  one  standard  deviation  and  is  approximately  42  percent  of 
the  full  width  at  half  maximum. 

x0  - y (FWHM),  y = (B  Jin  2)"2  * 0.42 

To  provide  generalized  dimensionless  variables  we  define  z s x/Xo  and 
define  A as  the  definite  integral  of  the  Gaussian. 

1 

A(a)  2 / e~2  dz 

i 


This  expression  can  be  numerically  approximated  by  an  inverted  polynomial 
series.  The  signal  can  then  be  written 

S =fZ B s(z)  dz  = 2S„  A (z  ), 

2B  b 

where  z„  is  the  half  signal  window  width.  An  experimentally  convenient 
way  to  express  the  background  is  as  counts  per  FWHM  interval.  We  have 
designated  this  Bo,  so  the  background  within  the  signal  window,  B,  is 

B = 2YB0Zg. 

This  reflects  the  assumption  that  the  background  is  flat.  Thus,  the 
signal  to  background  ratio 

_ S SD  A(zb) 

B yBo  zB 
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Now  with  these  expressions  for  S and  p»  <f>  becomes  a constant  times 
<t>',  a reduced  minimization  function. 

1 

<(>(z)  = -- 

/2  S0  p0/y 


where 


♦ *(z) 


2 

p =-S°9  a ~ 1 + & , 

0 Bo  

/aZ  + k /uZ  + p0A(z)/y 

"a(z) 


Note  that  all  variables  are  dimensionless  and  that  z is  a function  only 
of  the  three  input  parameters,  k,  p0,  and  3.  B 

3.  MODEL  SOLUTION.  The  complexity  of  the  expression  for  <f> ' doesn't 
lend  itself  to  an  analytic  solution  of  its  derivitives  to  determine  the  z 
corresponding  to  its  minima.  Thus  iterative  solution  by  computer  is  used. 
A program,  GRE0P,  has  been  written  to  run  on  the  MERDC  CDC  6600  to  calcu- 
late z . Some  of  these  results  are  summarized  in  Figure  3.  Here  z is 
not  displayed  - rather  Zg/y,  where  y = 0.42,  is  contoured  on  a po  versus 
k plot  in  which  the  background  is  assumed  well  known.  If  3 > 0 the  op- 
timal width  decreases  slightly.  Note  that  a window  width  approximately 
20  to  30  percent  above  the  signal  FWHM  is  in  many  cases  optimal.  The 
next  question  is  how  sensitive  is  the  choice  of  zg?  Figure  4 shows  that 
it  is  relatively  insensitive.  Changes  of  20  percent  induce  a decrease  in 
the  a's  of  less  than  2 percent.  Two  extreme  conditions  among  those  I have 
calculated  with  3=0  are  shown  and  are  quite  similar.  If  3 > 0 then  the 
trough  narrows  slightly.  Once  z^  is  determined  several  other  parameters 
can  be  calculated. 

Fraction  of  SD  within  window  = 2 A(z^) 

Optimal  window  width  as  fraction  of  FWHM  = 2zR/y 

4 

Relative  Gaussian  amplitude  at  window  boundary  = e 2 
The  actual  signal  to  background  ratio  within  the  window  is  p, 
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Figure  4 


Y ZB 

The  optimal  threshold  to  background  ratio,  t,  also  follows  after  a little 
arithmetic* 

T p + 1 + k A + p/a 


B 1 + k /T  + p/a 


These  dimensionless  ratios  are  shown  plotted  in  Figure  5 for  the  case 
in  which  3 - 0.  The  fehreshol d/background  ratio,  t,  is  plotted  versus 
the  measured  signal /background  ratio,  p.  A family  of  curves  can  be 
drawn  for  each  efficiency/specificity  ratio  parameter,  k.  The  input  pQ 
is  alee  shown*  As  e increases  the  optimal  threshold  also  increases,  as 
expected  in  order  to  move  it  further  from  the  less  well  known  background* 

Up  to  this  point  all  parameters  have  been  dimensionless  ratios.  If 
we  now  specify  SD,  Bc,  a , or  a - alee  dimensionless  but  not  independent 
numbers  since  p0  and  k h&ve  beefi  specified  - then  the  others  can  be  calcu- 
lated. For  example,  given  various  signal  levels,  So  counts,  corresponding 
to  expected  target  emission  intensities,  the  background  per  FWHM,  the  a's 
and  the  signal  and  background  within  the  window  follow. 

Bo  = S0/po 

V 1MV 

oe  = = k/*(zb) 

S = 2S0  A(zb) 

B = S/p  • 2So  zby/p0 

The  optimal  threshold,  false  signal  probability  due  to  statistical  fluc- 
tuations, and  the  target  detection  efficiency  can  now  be  calculated. 

T - B + Og/aB1  - tB 
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False  signal  probability:  1 per 

1/2  + A (a  ) 
s 

Target  detection  efficiency  = 1/2  + A(ae) 

4.  RESULTS.  Figure  6 is  a portion  of  the  printout  from  GRE0P  for 
the  case  in  which  3 = 1/2,  k - 1/2,  and  pG  = 1.  Those  variables  which 
were  previously  assigned  a symbol,  have  that  symbol  written  in  here.  Note 
particularly  the  false  signal  frequency  due  to  statistical  fluctuations 
and  the  detection  efficiency  as  they  vary  with  total  signal.  A signal  of 
200  counts  produces  almost  100  percent  efficiency  and  essentially  a com- 
plete lack  of  spurious  false  signals.  Such  rejection  is  surely  much  better 
than  the  false  signal  rate  due  to  target  simulation  in  most  scenarios. 

GRE0P  generates  a four  dimensional  matrix  of  conditions  spanning  some  of 
the  most  likely  values  of  the  input  parameters,  so  that  cross  comparisons 
can  be  performed. 


Several  related  problems  have  not  been  considered  in  this  paper,  such 
as  the  extension  of  this  analysis  to  non-quantized  input  and  the  problem 
of  collecting  data  continually  rather  than  processing  one  sample  at  a time. 
The  price  paid  to  achieve  continuous  converage  is  that  of  not  knowing  the 
time  domain  in  which  the  target  is  viewed,  thus  somewhat  higher  signal  count 
rates  are  required  to  yield  the  same  performance  as  with  a static  confi- 
guration. Other  modifications  such  as  multiple  targets  each  with  a distinct 
signature,  multiple  signals  from  a single  target  and  non-flat  backgrounds 
are  relatively  simple  extensions  of  this  analysis. 


5.  GAUSSIAN  RESPONSE  FUNCTION.  To  digress  a bit,  one  may  think  that 
a wei gh ti ng  f uncti on  wh i ch  match ed  the  response  function  would  provide  a 
better  window  to  filter  out  the  signal.  That  is  to  say  that  the  a's  would 
be  larger  if  the  window  were  Gaussian  weighted  than  if  weighted  by  a box 
function  as  we  have  done. 
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Notice  the  Gaussian  weight  is  not  normalized  to  unit  area  since  this 
would  unnecessarily  attenuate  the  signal.  A Gaussian  weighted  Gaussian 
signal  can  be  seen  here  to  yield  a detected  signal  of  71  percent  of  the 
total  signal  and  a signal  to  background  of  about  2/3,  see  Figure  7. 

Both  values  are  less  than  that  of  the  previously  presented  GRE0P  cal- 
culation,  86  percent  and  .69,  respectively.  Only  when  p6  and  k are  large 
and  a is  small,  does  p„  exceed  p,  which  drops  to  .55  under  the  worst 
conditions;  but  under  these  conditions  over  95  percent  of  the  signal  is 
processed.  Under  all  situations  calculated  the  processed  signal  is  lar- 
ger. Thus  if  there  is  a better  filtering  function  than  that  used,  it  is 
not  Gaussian  in  shape. 

6 . SUMMARY . Much  of  what  I have  discussed  is  related  to  communi- 
cation theory.  But  not  being  familiar  with  the  intricies  and  subtilies 
of  this  field  I have  started  from  scratch,  in  a detection  format,  assuming 
the  signal  is  digital  and  have  kept  the  approach  as  convenient  for  appli- 
cation as  possible.  The  input  parameters  follow  naturally  from  the  detec- 
tion conditions.  For  example,  3 (the  relative  background  determination 
error)  follows  from  the  detector  characteristics,  k (the  relative  impor- 
tance of  efficiency  to  specificity)  from  the  nature  of  the  detection  pro- 
blem and  pQ  (the  signal  to  background  ratio)  from  the  target  characteris- 
tics, The  optimal  threshold  can  then  be  determined  directly  since  it  is 
proportional  to  the  background  with  a proportionality  constant  x,  which 
can  be  calculated  once  3,  k and  p0  are  known. 

Another  practical  example  is  the  determination  of  the  minimum  signal 
in tensity, which  yields  the  desired  detection  characteristics  - that  is 
the  false  signal  rate  and  detection  efficiency.  Or  the  inverse  in  which 
a given  signal  intensity  is  available  and  it  is  desired  to  determine  how 
to  best  optimize  performance.  Given  the  range  or  parameter  values  which 
constrain  his  system*  the  designer  can  trade  off  one  for  the  other  to  ob- 
tain the  best  performance  with  the  weakest  signal.  Thus  if  nothing  else 
this  approach  to  target  visibility  and  detection  decision  optimization 
provides  the  desigher  with  an  organized  approach  to  improvement  of  his 
detection  system. 


Figure 


OPTIMIZING  A PRODUCTION  LINE 
FOR  COST  AND  QUANTITY 


Eileen  M.  R.  Weigand 
Manufacturing  Technology  Directorate 
Small  Caliber  Ammunition 
Modernization  and  Engineering  Division 
Frankford  Arsenal,  Philadelphia,  Pennsylvania 


ABSTRACT . At  various  times  in  the  production  life  of  any  manufacturing 
equipment,  it  is  desired  to  optimize  one  or  more  outputs  of  the  production 
machinery.  Such  outputs  might  be  cost  per  round,  number  of  pieces  produced 
per  minute  or  system  reliability.  Variable,  but  controllable,  factors  can 
influence  the  measured  outputs,  in  the  SCAMP  program,  the  operating  RPM  of 
the  equipment  and  the  length  of  the  running  time  between  repairs  and  pre- 
ventive maintenance  may  seriously  affect  the  outputs  stated  above. 

A statistically  designed  experiment  will  be  used  to  generate  mathemati- 
cal equations.  These  equations  will  then  be  used  to  predict  the  effects 
which  operating  rate  and  time  between  equipment  shut-down  will  have  on 
operating  cost  and  pieces  produced. 

The  results  will  be  used  to  establish  the  operating  and  maintenance 
policies  for  the  SCAMP  production  lines.  Since  each  output  may  be  optimized 
by  different  values  of  rates  and  time,  the  machinery  may  well  be  operated 
and  maintained  differently  in  peacetime  than  in  a state  of  full  mobilization. 

Response  surface  techniques  will  be  utilized  to  generate  the  mathemati- 
cal equations.  The  equations  will  then  be  used  to  establish  the  operating 
and  maintenance  policies  based  on  the  factor  to  be  optimized.  Moreover, 
the  response  surface  equations  should  then  be  used  throughout  the  life 
cycle  of  the  production  equipment  to  monitor  the  machinery  for  deviations 
from  the  original  conditions  which  may  indicate  a need  for  major  repair 
and/or  replacement. 

1 . INTRODUCT ION . At  the  present  time,  new  generation  equipment 
designed  to  automate  the  5.56mm  ammunition  production  line  is  now  entering 
its  final  stages  of  development.  The  equipment  consists  of  a group  of  sub- 
modules connected  in  series  which,  when  integrated,  shall  be  known  as  the 
"SCAMP”  production  module. 

The  Small  Caliber  Ammunition  Modernization  Program  consists  of  a 
Case  Submodule  which  manufactures  brass  cartridge  cases.  These  cases, 
in  turn,  are  fed  into  the  Primer  insert  Submodule  which  inserts  the  primer 
charge  into  the  base  of  the  brass  case.  At  the  same  time,  the  Bullet 


Submodule  manufactures  the  copper  bullet  which  is  mated  to  the  primed 
and  charged  brass  case  by  the  Load  and  Assemble  Submodule.  The  finished 
rounds  of  ammunition  are  then  prepared  for  shipping  by  the  Packaging 
Submodule.  The  entire  operation  is  completely  automated  and  computer 
controlled. 

The  submodules  are  composed  of  high  speed  rotary  turrets  linked 
by  a continuous  transfer  mechanism.  The  twenty- four  station  turrets 
are  designed  for  rapid  tool  changes  and  offline  repairs.  These  turrets 
are  also  designed  to  operate  at  a maximum  speed  of  fifty  revolutions  per 
minute.  The  cartridges  can  then  be  manufactured  and  assembled  at  rates 
approaching  twelve  hundred  rounds  per  minute. 

Op  to  now  the  submodules  have  been  tested  individually  to  debug  the 
equipment  and  to  fine  tune  the  various  steps  of  the  production  process. 
During  this  time,  each  of  the  prototype  submodules  has  displayed  its  own 
distinctive  operating  characteristics,  and  a data  bank  has  been  developed 
for  each  one.  We  know  the  range  of  speeds  over  which  the  individual 
submodules  can  operate,  approximately  how  and  when  the  individual  tool 
stations  on  each  turret  will  fail,  and  how  frequently  the  entire  submodule 
must  be  stopped  for  repairs  and  maintenance.  Turrets  and  transfer 
mechanisms  were  redesigned  to  increase  the  mean  time  between  failures  and 
to  reduce  the  repair  and  maintenance  downtime. 

The  individual  submodules  have  now  reached  the  stage  where  they  are 
ready  to  be  linked  into  a continuous  production  line.  At  this  point,  the 
thrust  of  the  SCAMP  statistical  research  shifts  to  the  module  as  an 
integrated  system.  In  particular,  the  emphasis  of  this  research  shall 
be  to  determine  how  well  the  equipment  meets  its  design  criteria,  and  then 
to  construct  and  formalize  operating  and  maintenance  routines  for  the  entire 
production  module.  Thus,  this  paper  will  present  one  technique  which  is 
under  consideration  to  optimize  both  the  cost  and  the  yield  of  this  highly 
complex,  computer  controlled  automated  production  line, 

2.  DEVELOPMENT  OF  THE  STATISTICAL  MODEL.  It  is  desired  that  the  pro- 
duction system  produce,  at  least,  a daily  average  of  384,000  acceptable 
quality  pieces  at  the  lowest  possible  unit  cost  during  an  eight  hour 
production  day.  Several  factors  of  the  production  module  affect  both  the 
number  of  good  pieces  produced  and  the  unit  cost.  Among  these  factors  are 
the  modular  operating  speed,  the  amount  of  time  the  module  operates  before 
it  is  shut  down  for  maintenance,  the  length  of  time  it  takes  to  perform 
maintenance,  the  quality  of  the  input  raw  materials,  lubrication,  and  the 
initial  settings  and  adjustments  of  the  tooling.  Of  these  factors,  some, 
such  as  lubrication,  are  not  easily  controlled.  Other  factors,  however, 
appear  to  exert  the  greatest  effect  on  submodule  productivity  and  cost. 

From  previous  tests  the  following  appear  most  likely.  They  are  the 
operating  speed  of  the  equipment  and  the  duration  of  time  the  production 
module  runs  before  it  is  shut  down  for  maintenance.  Fortunately,  they  are 
easily  controlled.  Thus,  we  shall  concentrate  on  this  latter  set. 


To  simplify  the  problem,  the  submodules  shall  be  considered  from  an 
integrated  system  perspective  only.  We  are  primarily  interested  in  the 
number  of  pieces  produced  and  their  unit  cost  when  the  module  is  run  for 
a specified  period  of  time  at  a predetermined  operating  rate.  Therefore, 
one  is  interested  in  determining  the  predictive  responses  of  (1)  cost 
and  (2)  quantity  for  a certain  range  of  values  of  speed  and  time.  This 
suggests  a multiple  polynomial  regression  approach  for  the  problem  resolution 
since  the  yield,  or  response,  may  be  perceived  as  a function  of  the  controlled 
variables.  (Figure  4)  This  function  is  called  the  response  surface.  If  the 
function  is  not  known,  it  can  sometimes  be  satisfactorily  approximated  within 
the  experimental  region  by  a polynomial  in  X.  Again  it  is  fortunate  that  we 
already  know  the  boundary  conditions.  For  the  operating  speed,  thirty  and 
fifty  revolutions  per  minute  are  the  constraints.  Moreover,  since  the 
production  schedule  is  eight  hours,  we  can  establish  reasonable  boundaries 
for  the  operating  time  of  the  equipment.  Let's  say  three  to  seven  hours. 

Once  again  prior  experience  with  the  machinery  has  limited  our 
experimental  region  to  the  areas  over  which  the  module  can  and  will  be 
operated.  Thus,  we  are  looking  for  the  optimum  response,  or  yield, 
within  the  above  constraints.  Moreover,  the  previous  data  lends  one 
to  believe  that  the  response  surface  will  be  adequately  approximated  by 
a quadratic  function.  (Figure  5)  Therefore,  the  experimental  points 
chosen  for  the  test  are  speeds  30,  40  and  SO  revolutions  per  minute 
(i.e.,  720,  960  and  1200  pieces  per  minute)  and  times  of  3,  5 and  7 
hours.  For  these  values  of  speed  and  time,  we  have  a 3^  factorial  design 
or  9 treatment  combinations.  Coding  the  minimum  values  as  -1,  the  maximum 
as  +1,  and  the  mid  value  as  0,  we  test  for  the  responses, 

Y^/  i ■ If-/  9,  at  the  nine  design  points,  (x^  , X2j)/j  = 1,  2,  3 (Figure  6), 

The  X matrix  for  this  model  is  shown  in  terms  of  the  coded  variables  in 
Figure  7.  However,  the  variance-covariance  matrix  for  this  model  is  not 
diagonal . 

< Since  a diagonal  variance-covariance  matrix  is  desired  to  eliminate 
any  interaction  between  the  coefficients  of  the  equation,  standard  tech- 
niques suggest  we  rewrite  the  response  surface  equation  as  shown  in  Figure  8 
to  obtain  a new  X matrix.  In  this  model  X2mi  is  the  mean  of  the  squared 
coded  variables  for  i ~ 1,2.  Now  the  variance-covariance  matrix,  shown  in 
Figure  9,  is  diagonal  and  we  can  determine  the  least  squares  estimates  for 
the  values  of  B by  the  matrix  equation  given  at  the  bottom  of  Figure  6. 

.These  values  of  B will  give  us  the  response  surface  equation.  Knowing 
the  estimates  for  the  regression  coefficients,  a regression  analysis 
of  variance  is  performed  to  determine  the  accuracy  of  the  response  surface 
equations  for  (1)  pieces  produced  and  (2)  cost. 

3.  ANALYSIS  OF  THE  FITTED  SURFACE.  Now,  after  we  have  determined 
the  proper  second  order  response  functions,  we  are  prepared  to  analyze  the 
fitted  surfaces.  The  maximum  point,  if  it  exists,  will  be  the  set  of 
conditions  such  that  the  first  partial  derivatives  are  simultaneously 
equal  to  zero.  This  set  of  conditions  is  called  the  stationary  point. 


nils  point,  however,  is  not  necessarily  that  which  maximized  the  response* 
In  fact  this  point  can  be  either  a maximum,  a minimum,  or  a saddle  point 
as  illustrated  in  Figure  10.  In  addition,  there  may  not  be  a point  at 
all,  but  some  type  of  a ridge  which  may  be  classified  as  rising,  falling 
or  stationary.  The  determination  of  the  nature  of  the  stationary  point, 
and  the  entire  response  surface,  is  the  ultimate  goal  of  the  experiment. 

The  analysis  begins  with  a translation  of  the  response  surface  to  the 
stationary  point,  X.  Then  the  response  function  is  expressed  in  terms  of 
new  variables  and  Z2  . This  corresponds  to  a rotation  of  the  axes  to 
correspond  to  the  principle  axes  of  the  contour  system.  The  form  of  the 
function  in  terms  of  the  Z variables  is  called  its  cononical  form.  Now, 
by  moving  along  the  new  axes  one  can  see  the  quickest  direction  to  travel 
to  find  the  maximum  or  the  minimum  responses. 

4.  OUTLOOK.  Slippages  in  schedules,  due  to  equipment  debugging  have 
delayed  the  integration  tests  needed  to  obtain  data  for  this  experiment. 
But  looking  at  the  preliminary  data  generated  by  the  submodules  we  can 
theorize  that  the  response  surface  for  the  number  of  pieces  produced  will 
be  a rising  ridge.  For  the  cost  surface,  we  expect  some  type  of  a basin. 
Thus,  the  conditions  under  which  we  will  operate  the  equipment  will  then 
be  somewhere  in  that  space  enclosed  by  the  two  response  surfaces.  If  the 
two  surfaces  do  not  intersect,  or  in  times  of  national  emergency,  then  we 
must  choose  whether  we  want  to  operate  for  maximum  yield  or  minimum  cost. 

Positive  results  from  the  operating  equipment  will  be  obtained  in 
approximately  six  months.  But  we  are  very  confident  that  these  techniques 
will  give  us  our  desired  results,  in  the  least  amount  of  time  and  for 
minimum  experimental  costs.  The  procedures  are  well-defined  and,  as  soon 
as  the  equipment  becomes  available,  we  can  generate  our  two  equations  to 
predict  the  number  of  acceptable  pieces  produced  and  the  unit  cost.  These 
equations  will  then  be  used  to  establish  the  optimum  operating  and 
maintenance  policies  for  the  SCAMP  production  line. 
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AN  APPLICATION  OF  THE  WE I BULL- GNEDENKO  DISTRIBUTION 
FUNCTION  FOR  GENERALIZING  CONDITIONAL  KILL  PROBABILITIES 
OF  SINGLE  FRAGMENT  IMPACTS  ON  TARGET  COMPONENTS 

William  P.  Johnson 
Vulnerability  Modeling  Branch 
Concepts  Analysis  Laboratory 
US  Army  Ballistic  Research  Laboratories 
Aberdeen  Proving  Ground,  Maryland  21005 


ABSTRACT.  It  has  previously  been  shown  in  a number  of  works  on  reli- 
ab i li ty  Wat~~the  Weibull- Gnedenko  distribution  gives  a good  description  of 
the  lifetime  of  numerous  elements  in  electronic  equipment  when  the  failure 
of  these  elements  is  regarded  as  the  exceeding  of  established  limits  and 
the  part  of  any  of  the  parameters. 

In  this  study  a modified  version  of  the  Weibull-Gnedenko  distribution 
has  been  used  to  establish  a relationship  between  the  conditional  proba- 
bility of  kill  of  target  components  (P^)  and  the  momentum  per  unit  area 

of  the  impacting  fragment  (MV/A) . A threshold  of  sensitivity  is  assumed 
for  each  component  resulting  in  a three  parameter  distribution  function. 
Techniques  have  been  developed  and  are  presented  which  allows  calculation 
of  approximate  values  for  each  of  the  three  parameters. 

1,  INTRODUCTION.  When  a high  explosive  munition  detonates,  the 
casing  that  surrounds  the  explosive  charge  is  fragmented  and  projected  out- 
ward. The  resultant  fragments  are  usually  irregular  in  shape,  vary  in 
weight  and  lose  velocity  through  air  at  a rate  that  is  proportional  to 
several  physical  parameters.  Equipment  and/or  weapon  systems  struck  by 
these  fragments  may  be  unharmed,  incapacitated,  or  killed.  For  this 
reason,  a portion  of  the  Army’s  defense  effort  is  devoted  to  determining 
the  relationship  between  parameters  of  the  impacting  fragments  and  the 
resultant  system  damage.  An  important  part  of  this  effort  has  been  the 
establishment  of  conditional  kill  probability  curves. 

Conditional  kill  probabilities,  for  critical  target  components , have 
been  developed  as  a function  of  the  striking  mass  and  selected  velocities 
of  the  striking  fragment.*  The  impracticality  of  using  these  same  tech- 
niques to  make  measurements  over  the  entire  spectrum  of  fragment  mass- 
velocity  combinations,  makes  a generalized  model  a necessity.  Models  of 
this  type  facilitate  the  vulnerability  analyses  designed  to  rate  the  capa- 
bilities of  existing  and  prospective  systems  on  the  basis  of  their  abilities 
to  withstand  impacts  from  fragments  or  shaped  charges. 


V E.  Kinsler,  "Conditional  Kill  Probabilities  for  Single  Fragment  Impacts 
on  Components  of  the  Soviet  KRA2-214  Truck  (U),"  Ballistic  Research  Labo- 
ratories Memorandum  Report  No.  1995,  July  1969,  DDC  AD504240L,  CONFIDENTIAL 
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KILL  PROBABILITY  CP, 


Figure  1 - Typical  Probability  Function  for  Fragments  Striking  a Hypo 
thetical  Component. 


Previous  attempts  to  relate  fragment  size  to  conditional  kill  proba- 
bilities used  step  functions.  The  use  of  these  functions  required  the 
user  to  interpolate  between  curves  for  masses  which  were  not  included  in 
the  original  analysis  and  between  points  of  a given  curve  for  velocities 
not  included.  Computer  programs  using  these  curves  required  an  extensive 
amount  of  memory  in  order  to  accommodate  all  of  the  input  data  normally 
required  for  a detailed  analysis.  Computer  run  time  was  usually  extensive, 
due  in  part  to  the  numerous  conditional  statements  required  to  determine 
whether  the  right  combination  of  curves  was  being  used  in  the  interpola- 
tion process.  Figure  1 illustrates  a typical  step  function  representation 
of  the  experimental  data.  It  should  be  kept  in  mind  that  this  approach 
required  a curve  of  this  type  for  each  mass  included  in  the  sample 
population. 

The  model  outlined  in  the  subsequent  pages  of  this  report  was  developed 
to  overcome  the  limitations  in  the  step  function  approach.  The  primary 
aim  was  to  develop  a mathematical  relationship  which  would  satisfactorily 
represent  the  data  points  and  would  expedite  subsequent  computer  analysis 
of  the  vulnerability  of  the  target.  The  accuracy  of  the  data  points  them- 
selves were  not  questioned,  but  merely  accepted  as  the  best  available 
representation  of  the  true  data  points.  A description  of  the  technique 
used  to  develop  the  experimental  data  may  be  found  in  references  2 and  3. 

2.  PROCEDURE.  The  experimental  data  inlcuded  conditional  kill 
probabilities  as  a function  of  kill  criterion,  fragment  mass,  fragment 
velocity  and  attack  orientation.  Of  these  four  variables,  kill  type  and 
fragment  attack  orientation  were  held  constant  for  each  analysis.  More 
specifically,  only  the  random  attack  condition  was  used  to  develop  the 
model.  The  random  condition  assumes  that  attack  of  the  target  is  equally 
likely  from  all  directions.  The  same  procedures  used  to  establish  the 
model  could  also  be  used  to  determine  regression  constants  that  would 
provide  conditional  kill  probabilities  as  a function  of  any  other  attack 
orientation  or  kill  criterion. 

The  weights  of  the  fragments  included  in  the  experimental  data  were 
the  following: 


(grains) 

Weight 

(kilograms) 

1 
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Weight  (grains) 
30 
60 
120 
240 
500 
1000 
2000 


Weight  (kilograms) 
1.94  x 10"3 
3.89  x 10"  3 
7.78  x 10~3 
1 . 56  x 10"2 
3.24  x 10"2 
6.48  x 10“2 
1.30  x lCT1 


Striking  velocities  ranged  from  91.44  m/s  to  2133.6  m/s  (300  to  7000 
fps.) . 


Independent  plots  were  made  for  several  of  the  target  components. 
Several  conditional  probability  of  kill  - abscissa  combinations 


were  considered  before  it  appeared  that  a relationship  between  the  condi- 
tional probabilities  and  the  ratio  of  the  momentum  of  the  fragment  to  its 
average  presented  area  existed. 


The  root  mean  square  deviation  (erms)  of  the  experimental  data  points 
from  the  regression  curve  was  used  as  the  criterion  for  selecting  the  ana- 
lytical model  which  generally  best  described  the  data  population.  Erms 
was  calculated  from  the  following  equation: 


where: 


SS-3 


= predicted  conditional  kill  probability 
- observed  conditional  kill  probability 
= sample  size. 


CD 


3.  THE  WEIBULL  - GNEDENKO  DISTRIBUTION.^  Let  us  consider  a system  consisting 
of  a group  of  elements  and  possessing  the  following  properties:  (1)  The 


Gertsbakh,  I.  8.,  Kordonsky,  Kh.  B,,  ’’Models  of  Failure,”  Springer-Verlag, 
New  York,  1969 
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failures  of  the  elements  are  mutually  independent.  (2)  Failure  of  any 
element  is  treated  as  a failure  of  the  entire  system.  We  call  such  systems 
chain  systems.  Let  a denote  the  lifetime  of  the  chain  system  and  let  a* 
denote  the  lifetime  of  the  ith  element  of  the  system  for  i - 1,  2,  3,... 2. 
In  such  a case: 

12  3 Z 

a = min  (a  , a , of  ....  a ) 

We  point  out  an  important  special  case  suppose  that  all  elements 
of  the  chain  system  have  an  exponential  distribution  of  the  lifetime. 

In  accordance  with  formulas  derived  in  reference  4,  the  distribution 
function  F(t)  is  equal  to: 

Z 

-£  ah 

F(t)  = 1 - e 1=1 

where  the  a1  are  parameters  of  the  distribution  of  the  elements  of  the 
chain  system. 

Of  special  interest  is  the  situation  generalizing  the  case  just 
described  but  having  the  following  features.  The  number,  Z,  of  elements 
of  the  chain  system  is  great  and  all  the  distribution  functions  F. (t) 
are  such  that:  1 

Fi(t)  = gtA  + 0(t)X  (2) 

Where  g and  X are  positive,  as  t * 0. 

This  relationship  determines  the  order  of  the  infinitesimal  F(t) 
for  small  t.  One  can  show  that,  for  large  Z,  the  distribution  function 
F(t)  is  well  approximated  by  an  expression  of  the  form: 


F(t)  = {l  - exp[-6(t)X]  ; ^0°  (3) 

This  distribution  was  proposed  by  W.  Weibull  in  1939  without  math- 
ematical foundation.  A rigorous  mathematical  treatment  of  related  prob- 
lems was  done  by  B.  V.  Cnedenko  in  1941 

Equation  3 shall  be  called  the  Weibull-Cnedenko  Distribution  through- 
out this  report. 

It  has  been  shown  in  a number  of  works  on  reliability  that  a 
We ibul 1-Gnedenko  distribution  gives  a good  description  of  the  distribu- 
tion of  the  lifetime  of  numerous  elements  in  radio-electric  equipment 


Bolotin,  V.  V.,  "Statistical  Methods  in  Construction  Mechanics," 
Stroyizdat,  1965 
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when  the  failure  of  those  elements  is  regarded  as  the  exceeding  of 
established  limits  on  the  part  of  any  of  the  parameters. 

Often  there  are  situations  in  which  there  is  a threshold  of  sensi- 
tivity, t , that  leads  to  a displacement  of  the  distribution.  For  these 
situations  the  Weibull-Gnedenko  Distribution  becomes: 

F(tj  1 - to  (4) 

}o  , t < t 

Where  t is  defined  as  the  threshold  of  sensitivity.  The  defini- 
o 

tion  originated  in  metrology.  Its  significance,  is  that  until  the  para- 
meter t exceeds  or  equals  the  threshold  of  sensitivity  the  device  under 
investigation  does  not  "feel"  the  effect  of  the  load  and  it  is  only  when 
t _>  t that  this  influence  becomes  perceptible  and  causes  a probability 

of  failure. 


APPLICATION  OF  WEIBULL-GNEDENKO  DISTRIBUTION  AND  CALCULATION  OF 


Gnedenko  distribution  was  found  to  represent  the  experimental  data  popu- 
lation extremely  well. 


This  variation  assumes  the  following  form: 

f _ e-B(MV/A  - K)N]  , MV/A  > K 
rmax  *■ 

(5) 

0 , MV/A  <_  K 

Where  the  combination  MV/A  is  substituted  for  the  parameter  t in  the 

original  distribution,  and  K becomes  the  threshold  of  sensitivity.  In 

addition  the  distribution  is  multiplied  by  a constant  P ♦ This  multi- 

r J max 

plication  factor  was  included  so  as  to  provide  constraints  on  the  regression 
curve  such  that  0 < P„  ...  < P in  contrast  to  the  constraints  provided 

by  the  original  distribution  0 F(t)  < 1.0.  These  new  constraints ’were 
prompted  by  practical  consideration  which  indicate  that  the  maximum  proba- 
bility of  kill  that  can  be  obtained  on  certain  components  is  less  than  1. 


An  explanation  of  the  terms  included  in  the  above  equation  follows: 


Pj^j  - represents  the  conditional  kill  probability 
M - represents  the  weight  of  the  fragment  (kg) 


160 


V - represents  the  striking  velocity  of  the  fragment  upon  the 
component  (m/s) 

2 

A - represents  the  average  presented  area  of  the  fragment  (cm  ) 

P - Maximum  value  of  P„  /IT  in  experimental  data  set 
max  K/H  r 

e - represents  the  base  of  the  natural  logarithm 

B § N are  regression  constants. 

In  the  above  equation,  K dictates  the  MV/A  value  at  which  the  pre- 
dicted kill  probability  diverges  from  zero.  In  order  to  determine  this 
constant  a computer  program  was  developed  in  which  cutoff  points  were 
selected  at  small  intervals  between  zero  and  the  smallest  ratio  of  MV/A 
available  for  the  component  - kill  type  combinations  under  consideration. 

A combination  of  the  constants  B § N were  calculated  with  each  selection 
of  K,  from  the  solution  of  normal  equations  for  a straight  line,  after  a 
double  logarithmic  transformation  had  been  made  on  equation  5.  The  root 
mean  square  error  was  calculated  from  each  combination  of  B,  N £ K by  using 
the  calculated  values  of  these  constants  in  Equation  5.  It  should  be 
noted  that  the  erms  was  determined  thru  the  use  of  Equation  5 and  not  thru 
its  logarithmically  transformed  version.  The  logarithmic  transformation 
served  solely  as  a means  of  obtaining  values  of  the  constants. 


Equation  1 was  used  to  determine  the  average  deviation  of  the 
regression  line  from  the  data  population.  The  combination  B,  N S K 
selected  to  be  used  was  that  group  which  minimized  erms. 


A second  equation  was  used  to  establish  95  percent  confidence  limits 
on  the  individual  values  of  for  a second  value  of  MV/A.  This  equa- 

tion assumed  the  following  form: 


K/H 


~ t 


.025*  (SS-3)  P 


± PK/H  I 


P 

h K/H 


+ 1 .025  (SS-3) SP 


(6) 


where:  PR/l[ 

SS 


s 


P 


represents  the  conditional  kill  probability 
represents  the  sample  size 

represents  the  standard  deviation  of  the  P^/n 
population 


.025  - represents  the  value  taken  from  the  students  t 
table  with  (SS-3)  degrees  of  freedom. 


Confidence  intervals  of  this  form  have  a minimum  at  the  mean  of 
the  independent  variable  values  included  in  the  experimental  data  set. 


l6l 


Therefore,  it  is  expected  that  predictions  made  on  either  extreme  of  the 
data  set  would  have  larger  confidence  intervals.  In  these  analyses  these 
intervals  were  restricted  to  the  interval  0 < P„.„  < 1,  although  mathe- 

— K/H  — 


matically,  they  would  have  fallen  outside  of  these  intervals.  This  restric- 
tion forces  the  confidence  intervals  to  reach  a plateau  at  each  of  these 
limits  in  some  instances.  Figure  2 illustrates  the  goodness  of  fit  of  the 
above  model  to  data  from  a typical  component. 


5.  MODEL  VALIDATION.  Upon  completion  of  the  program  development,  constants 
were  calculated  £or  several  data  sets,  and  given  to  vulnerability  analysts 
to  test  in  established  vulnerability  programs.  The  purpose  was  to  deter- 
mine whether  the  use  of  the  equation  rather  than  the  step- functions  would  in 
fact  decrease  the  running  time  of  the  programs  as  had  been  speculated  and 
also  to  determine  whether  significant  statistical  differences  existed 
between  the  results  of  the  two  methods. 


It  was  observed  that  the  vulnerable  area  calculations  based  on  mass- 
velocity  combinations  contained  within  the  original  data  set  were  reasonably 
close  to  values  obtained  thru  the  use  of  step- functions,  however,  vulnerable 
area  values  determined  from  extrapolated  values  of  the  data  set  were  gen- 
erally larger. 


A plot  of  the  Pv/„  data  from  a typical  data  set  revealed  that  the 


K/H 

step- functions  and  the  generalized  curve  diverged  at  the  maximum  P 


K/H 


value  for  each  of  the  fragment  masses.  As  can  be  observed  in  Figure  3, 
the  step  function  approach  assumes  that  a maximum  P^^  can  be  achieved 

and  the  curve  will  remain  constant  at  this  value.  But  because  the 

Weibul 1-Gnedenko  distribution  provides  a curve  with  values  directly 

proportional  to  the  independent  variable  MV/A,  P , values  greater  than 
those  obtained  in  the  experimental  data  set  were  obtained  for  each  mass 
smaller  than  the  largest. 


To  circumvent  this  problem  it  was  decided  that  a second  equation  was 
needed  which  would  enable  the  analyst  to  determine  the  maximum  P^jj  value 

obtainable  for  a given  component  as  a function  of  a fragments  mass.  The 
logic  was  that  during  the  actual  calculation  of  vulnerable  areas  in  the 
analyst's  computer  programs,  a comparison  could  be  made  between  the  P . 


value  determined  from  the  generalized  curve  and  the  maximum  P 


K/H 


value 


which  could  be  obtained  from  the  fragments  mass.  The  smaller  of  these 
two  values  would  then  be  used  in  the  vulnerable  area  calculations. 
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To  carry  out  this  logic  a relationship  between  the  maximum 

values  for  each  mass  contained  in  the  experimental  data  set  and  the  mass 
itself  was  needed*  Interpolation  of  the  maximum  value  of  P^^  for  masses 

not  included  in  the  data  set  was  the  main  reason  for  requiring  a gener- 
alized relationship. 

As  had  been  the  case  earlier,  it  appeared  that  the  Weibull-Gnedenko 
Distribution  provided  the  best  relationship  between  the  variable  of  interest. 
The  relationship: 


P = 1 - 

max 


-B1 


(log10M  + 1)  - K1 


N1 


(7) 


gave  excellent  regression  fits  of  the  experimental  data  with  small  erms 
values  as  determined  by  Equation  1 (Figure  4) . 

In  the  above  equations  the  independent  variable  can  be  chosen  as 
(log^M)  if  the  masses  of  interest  are  greater  than  1 unit.  The  above 


form  will  provide  P 


K/H 


values  for  fragments  with  mass  >0.1  units. 


max 


The  actual  unit  of  mass  chosen  to  use  in  the  equation  is  immaterial  since 
the  values  of  Bl,  N1  and  K1  will  adjust  to  the  unit  used. 

Upon  completion  of  the  generalization  of  the  maximum  as  a 

function  of  each  fragment's  mass,  the  idea  to  use  the  P value  pre- 

h * max  r 

dieted  from  Equation  (7)  as  a variable  in  Equation  (5)  evolved. 


The  objective  was  to  provide  an  upper  limit  on  the  P^^  predictions 

for  each  mass  under  consideration  and  to  avoid  the  comparisons  required 
in  the  earlier  method.  To  determine  whether  this  was  a feasible  tech- 
nique, both  sides  of  Equation  (5)  were  divided  by  P giving: 

1113.X 


^K/H  / ^max 


± _ e-B  (MV/A  - K)N 


(8) 


Equation  (8)  indicates  that  the  distribution  function  for  the  ratio  of 

the  conditional  probability  of  kill  to  the  maximum  probability  of  kill 

for  a given  fragment  is  given  by  the  We ibu 11 -Gnedenko  equation.  All 

variables  in  Equation  (8)  are  identical  to  those  described  for  Equation 

(5)  with  the  exception  of  P . Pmnv  in  the  above  equation  is  the  maximum 

max  max 


MASS 


STRIKING  VELOCITY  x 10  W») 

Figure  3*  Divergence  of  Model  and  Step  Functions 


probability  of  kill  obtainable  against  a selected  component  for  the 

desired  fragment  weight.  Since  P is  determined  from  Equation  (7), 

max 

Equation  (7)  must  be  developed  prior  to  using  Equation  (8) . 


In  studies  designed  to  document  the  prediction  error  of  P 


K/H 


values 


obtained  from  Equation  (8)  and  those  obtained  from  Equations  (5)  and  (7) , 
both  methods  appeared  to  work  equally  well.  To  illustrate  this  point 
nine  components  are  listed  below  with  associated  root  mean  square  errors 
(erms)  obtained  from  both  methods: 


COMPONENT  NUMBER 

ERMS 

EQUATION  5,  7 

ERMS 

EQUATION  7,8 

1 

.072 

.100 

2 

.003 

.004 

3 

.013 

.008 

4 

.000 

.001 

5 

.052 

.027 

6 

.069 

.042 

7 

. 166 

.192 

8 

.137 

.120 

9 

.110 

.110 

Although  there  appears  to  be  little  difference  in  the  erms  values 
calculated  from  each  of  the  two  methods.  Equation  (8)  may  be  easier  to 
use  during  actual  vulnerability  calculations  and  is  therefore  recommended. 
Both  methods  prevent  the  divergence  of  the  step- function  and  the  gener- 
alized curves  illustrated  previously,  and  both  appear  to  reduce  computer 
running  time  over  that  which  was  previously  required. 

Though  it  appears  that  the  objectives  of  the  generalization  have 
been  achieved,  more  formal  tests  are  needed  to  determine  whether  the 
resulting  vulnerable  area  values  produced  by  (1)  the  step- function  curves 
and  (2)  the  generalized  equation  are  statistically  equivalent.  Suggested 
techniques  are  discussed  in  the  next  section. 

6.  STATI  ST  I CAL  N1ET1 IQDS . ^ ^ In  order  to  determine  whether  there  is  a 
s i gn i fie  an  t s t at is  t i c a 1 difference  between  the  -vulnerable  area  values 
obtained  with  the  step- functions  and  the  values  obtained  with  the  gener- 
alized equations,  the  following  hypothesis,  level  of  significance  and 
test  statistics  are  suggested. 


^Dixon,  Wilfrid  J.  and  Massey,  Frank  J.  Jr.,  "Introduction  to  Statistical 
Analysis,"  2nd  Edition,  McGraw-Hill  Book  Company,  Inc.,  1957,  Chapters 
9 § 14. 

7 

D.  V.  Huntsberger,  "Elements  of  Statistical  Inference,"  Second  Edition, 
Allyn  § Bacon,  1967,  Chapter  2. 
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MAXIMUM  PK/H 


SAMPLE  COMPONENT 

ERMS*  0.02  Pmax’0'64 

Bl*  0.3073  Nl=  2.0545 
SUMN  ■ 10.  K1»0. 


LOG  jo  (MASS) 


Figure  4 * Maximum 


Versus  Fragment  Weight 
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1,  Let:  X be  the  vulnerable  area  value  obtained  from  mass  (X) 

S and  velocity  (Y)  using  the  step-function  approach. 

X be  the  vulnerable  area  value  obtained  from  mass  (X) 

® and  velocity  (Y)  using  the  generalized  equation 
approach . 

Let  the  difference  between  the  two  vulnerable  area  values  previously 

defined  above  equal  X where  X = X - X . Clearly,  X will  only  be  equal 

g 

to  0,  that  is  no  difference  between  the  individual  values  exists  when 
Xs  = Xg  otherwise  X will  be  either  a positive  or  a negative  number 

(X  > 0 or  X < 0), 

Let  SS  represent  the  number  of  differences  to  be  considered. 


2.  Test  the  hypothesis  that  the  mean  difference  between  the  vul- 
nerable area  values  generated  by  step-functions  (y^)  and  the  vulnerable 

area  values  generated  by  the  generalized  equations  (y  ) is  statis- 

tically equivalent  to  zero,  (i.e.  y^  = yg  - y^  ^ EX/SS  = 0). 


3.  A test  of  significance  is  in  general  terms  a calculation  by 
which  the  sample  results  are  used  to  throw  light  on  the  truth  or  falsity 
of  a null  hypothesis.  A quantity  called  a test  statistic  is  computed, 
which  measures  the  extent  to  which  the  sample  departs  from  the  null 
hypothesis  in  some  relavant  aspect.  If  the  value  of  the  test  criterion 
falls  beyond  certain  limits  into  a region  of  rejection  the  departure  is 
said  to  be  statistically  significant  or  more  concisely  significant.  Tests 
of  significance  have  the  property  that  if  the  null  hypothesis  is  true 
(not  difference  between  means)  the  probability  of  obtaining  a significant 
result  has  a known  value  most  commonly  referred  to  as  a and  chosen  as 
0.05  or  0.01.  This  probability  is  the  significance  level  of  the  test. 


For  the  purpose  of  the  analysis  one  should  choose: 

4.  a - .05  or  a - .01.  The  test  criterion  should  be: 


S/v'fTS  s/f§<r  S//SS 
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where:  Xg  = Vulnerable  area  estimate  from  step- functions 

= Vulnerable  area  estimate  from  generalized  curve 
SS  = Sample  size 

S = estimate  of  population  standard  deviation  and  is 
computed  from  the  following: 


6.  Our  population  should  be  large  enough  so  that  the  central  limit 
theorem  is  applicable.  Therefore,  our  test  statistic  will  have  a t dis- 
tribution with  SS-1  degrees-of-freedom. 

7.  Our  rejection  region  defined  in  paragraph  5 will  be  obtained 
from  standard  tabular  values  of  the  t distribution  with  SS-1  degrees- 
of-freedom. 

In  other  words  we  will  only  be  able  to  statistically  state  that 
vulnerable  area  values  computed  from  the  two^equations  are  different 
if  the  computed  value  of  the  test  criterion  t is  outside  of  the  region 
defined  for  the  combination,  that  is: 

I tl  s-  tl/2a,  SS-1. 

Otherwise,  we  will  have  to  say  that  there  is  not  enough  evidence  to 
reject  the  hypothesis. 

An  alternate  approach  to  the  above  would  be  the  use  of  a two  way 
analysis  of  variance  which  would  offer  the  advantage  of  determining 
whether  the  differences,  if  any  exist,  are  between  the  row  effects 
(masses) , column  effects  (velocities)  or  both. 

7.  SUMMARY.  The  We ibull -Gnedenko  Distribution  function  has  been  used 
to  establish  a relationship  between  the  conditional  probability  of  kill 
(P^,/jj)  of  a fragment  and  the  fragments  momentum  per  unit  area  (MV/A). 

Statistical  tests  are  suggested  which  determine  whether  a significant 
statistical  difference  exists  between  vulnerable  area  values  calculated 
using  this  distribution  function  and  vulnerable  area  values  calculated 
from  currently  used  step- functions . 
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If  it  is  determined  that  no  significant  difference  exists,  then  it 
is  recommended  that  this  distribution  function  be  used  in  subsequent 
vulnerability  analyses  because  of  its  convenience  and  the  reduction 
obtained  in  computer  running  time. 


DECISION  THEORY  APPROACH  TO  GRADING 
BINOMIAL  POPULATIONS 

Paul  Williams 

U S Army  Materiel  Systems  Analysis  Activity 
Aberdeen  Proving  Ground , Maryland 


ABSTRACT . A method  of  classifying  lots  of  mass  produced  articles 
into  one  of  k categories  on  the  basis  of  the  number  of  defectives 
allowable  in  a single  sample  size  n (n<k-l)  is  presented.  A beta 
distribution  is  assumed  as  the  prior  distribution  of  the  true  lot 
fraction  defective  where  the  parameters  are  to  be  estimated  using 
knowledge  obtained  from  lots  inspected  in  the  past.  An  optimum 
decision  rule  for  obtaining  the  allowable  number  of  defectives  in 
a lot  is  developed  where  the  allowable  lot  fraction  defective  and 
sample  size  are  determined  prior  to  testing. 

1.  INTRODUCTION.  The  production  of  large  quantities  of 
mass  produced  artilces  often  necessitates  dividing  the  articles , 
for  the  purpose  of  homogeneity,  into  groups  called  lots.  It  is 
then  desirable  for  the  producer  or  consumer  to  place  the  lot  in  a 
category  based  on  the  quality  or  reliability  determined  from  some 
characteristic  of  the  individual  articles  in  the  lot.  If  the 
articles  in  the  lot  are  to  be  labeled  effective  or  defective  after 
inspection,  then  the  inspection  is  to  be  by  attributes  since 
inspection  by  variables  is  based  upon  quantitative  measurements. 

Where  large  lots  are  concerned,  the  cost  of  testing  of  each 
item  may  be  excessive  or  where  the  item  is  destroyed  by  testing, 
a sampling  plan  to  estimate  the  number  of  defectives  in  the  lot 
must  be  devised.  Such  a plan  is  the  single  sampling  plan  where 
a sample  of  n items  is  selected  from  a large  lot  of  size  N.  After 
each  item  has  been  inspected  or  tested,  the  number  of  defective 
items  (r)  is  determined  and  the  lot  is  placed  into  one  of  k categories 
based  on  the  number  of  defectives  in  the  sample. 

A basic  solution  to  the  single  sampling  problem  is  to  decide 
on  an  acceptable  quality  level  p^  such  that  the  consumer  desires 

to  accept  almost  all  lots  of  fraction  defective  p^  or  less  and 

also  to  specify  an  objectionable  quality  level  p^  which  represents 

lots  of  quality  so  inferior  that  the  consumer  cannot  accept  more 
than  a few  lots  of  this  quality.  By  specifying  quality  levels 
and  p^,  the  risk  a of  rejecting  a lot  of  lot  fraction  defective 

Pl  and  the  risk  & of  accepting  a lot  of  objectionable  quality  can 


171 


be  determined  given  the  distribution  of  the  process  sampled.  The 
problem  then  is  to  find  the  smallest  sample  size  and  acceptance 
number  which  will  give  the  desired  protection.  In  the  case  of  fixed 
sample  size,  the  problem  is  to  determine  a minimum  acceptance  number. 

The  sampling  plan  considered  in  this  paper  is  the  single 
sampling  plan  and  is  solved  using  the  decision  theory  approach, 
that  is,  a priori  information  will  be  combined  with  data  from  the 
sampling  program.  However,  economic  considerations  are  assumed 
to  be  unimportant  or  small  when  compared  to  the  risk  of  making  a 
wrong  decision, 

2.  DEFINITIONS  OF  THE  ACTION  SPACE.  The  sampling  plan  then 
calls  for  obtaining  a fixed  sample  of  n items  from  the  lot  and  test- 
ing to  determine  the  number  of  items  (r)  which  are  defective.  If 
the  lot  contains  c or  less  defectives,  the  lot  is  accepted,  but 
if  the  lot  contains  c+1  defectives  the  lot  is  rejected.  With  any 
sampling  plan  there  is  the  possibility  of  making  a wrong  decision, 
that  is,  taking  an  action  which  would  not  be  taken  if  the  true 
quality  of  the  lot  were  known  before  making  a decision.  The  set  of 
all  possible  actions  which  may  be  taken  to  solve  a problem  is  known 
as  the  action  space.  As  an  example,  the  space  of  possible  actions 
could  for  a decision  about  the  disposition  of  a lot  contain  two 
points;  (1)  accept  the  lot  or  (2)  reject  the  lot.  The  correct  action 
to  take  would  depend  on  the  true  state  of  nature;  that  is  the  actual 
proportion  of  defective  items  in  the  lot.  The  difficulty  of  course 
is  that  the  true  state  of  nature  is  not  known  unless  the  entire  lot 
is  tested. 

The  action  space  can  then  be  defined  as  placing  a lot  into 
one  of  k categories.  A lot  is  placed  into  one  of  k categories  by 
using  the  following  rule: 

if  0 j<  r s ci  Lot  is  Grade  A 

Cj  + 1 <_  r _<  C£  Lot  is  Grade  B 


ck-i  + 1 <.  r _<  cr  Lot  Grade  k 

Each  of  the  above  intervals  should  be  determined  from  the 
allowable  lot  fraction  defective  for  each  category. 
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3.  PRIOR  DISTRIBUTION.  In  considering  a production  lot 
in  which  the  items  can  be  classified  into  two  groups,  effective 
and  defective,  a natural  probability  model  is  the  Bemolli  process 
which  has  probability  mass  function: 

(1)  f(x)  = px  Op)1'*  x = o Success 

x = 1 Failure 


where  p is  the  probability  that  a randomly  chosen  item  in  the 
lot  will  be  defective.  As  the  prior  distribution  of  the  parameter 
p in  the  Bemolli  process,  we  choose  a Beta  Distribution  of  the 
form 


(2) 


f(p;AB) 


r (a+b)  pA-1  (l-p)6'1 

rCA)r(B) 


where:  o < p < 1 

A > o 


B > o 


where  A and  B are  to  be  estimated  from  prior  knowledge,  that  is 
previous  testing.  The  Beta  Distribution  has  been  chosen  as  a 
prior  distribution  for  the  true  defect  rate  in  the  lot  for  two 
reasons:  (1)  The  random  variable  p in  the  Beta  Distribution  is 

defined  in  the  interval  o < p _<  1 as  is  the  parameter  in  the 
Bernolli  process.  (2)  The  Beta  Distribution  represents  a rich 
family  of  possible  densities  to  express  our  knowledge  of  prior 
information. 

For  various  values  of  A and  B we  can  generate  the  family 
of  Beta  Distributions,  some  examples  of  which  are  given  in 
Figure  1.  To  select  a member  of  the  family  of  Beta  Distributions 
we  can  choose  its  mean  and  variance  as  an  expression  of  our  prior 
beliefs  about  the  unknown  parameter  p.  The  first  and  second  moments 
about  the  origin  are: 


(3) 


E Cp)  - 


A 

A+B 


(4) 


E (p2)  = y 1 X ft— (A+_U 

lP  J y2  (A+B)  (A+B+l) 


Using  the  number  of  defects  as  observations  on  p,  parameter 
estimates  for  the  Beta  can  be  estimated  by: 
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FIGURE  1 

Beta  Distributions 


fO.I  0,2  0.3  0.4  0.5  0.6  0.7  0.8 


A1 

A yl  ^yl  _ y 2^ 

(5)  A = T! — 7 — 

»i  C«2  ' V * y2  (1  - V 

- 

(y-i  ■ p t (i  ■ yJ 

(6)  B = — i 7 71 — 

ui  Cy  2 ” yP  + w 2 ^ ~ yi^ 

In  some  actual  problems  the  choice  of  a specific  beta  prior 
distribution  may  be  largely  subjective  at  the  start  of  production. 
However,  after  a few  lots  have  been  produced  the  selection  of 
the  prior  distribution  becomes  more  objective  by  incorporating 
previous  lot  results . 

4,  PROBABILITY  OF  CORRECTLY  GRADING  A LOT.  Under  the 
plan  stated  in  paragraph  3 and  using  the  formula  of  total 
probability*,  the  probability  of  correctly  grading  the  lot  for 
the  two  action  state  case  is  the  sum  of  the  probabilities  of 
accepting  lots  of  true  quality  p^  and  rejecting  lots  of  true 

quality  p^*  This  probability  can  be  expressed  by  an  equation 

of  the  form: 

Probability  of  Correctly  Grading  Lot  = P 
- Pr  (o  <_  p < px)  Pr  (o  ^ r <_  c^o  _<  p < px) 

+ Pr  (p:  <_  p < p2)  Pr  (cx  + 1 c <_  c2|p1  <_  p < p2) 


Using  the  distribution  assumptions,  the  probability  of  correctly 
grading  k categories  in  the  above  expression  can  be  written  as: 


*B.V.  Gredenko  "Theory  of  Probability" 
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. . + 


(A+B-l)!  pA~X  (l-p)®"1 
(A- 1)  ! (B- 1) ! 


(A+B-n-i  pa-1  (l-p)8'1 

(A-l) ! (B-l) ! 


r=c2+l 


c 


1 


E 


(”)Pr  (i-P)n'rdP 

frV  (l-p)n"r  dp 


The  problem  then  is  to  determine  the  number  of  defectives  C2C2C3 
...  which  will  make  placing  a lot  into  one  of  k categories  an 
optimum  act  under  uncertainty. 

5.  TERMINAL  ANALYSIS.  The  first  step  in  terminal  analysis 
is  to  determine  the  posterior  distribution  of  the  process.  If  we 
let  %2  X3  . ...X^  represent  a random  sample  from  the  lot,  the 

distribution  of  any  x given  p can  be  written: 


f (x/p)  = pX  (l-p)1~X 
x = 0,  1 


Since  the  observations  are  independent  the  conditional  density  of 
all  the  X's  in  the  sample  is: 


(9) 


f (XrX2,X3  ...  Xn|p)  - 


To  simplify  the  problem,  the  n - tuple  X2  ...  Xn)  is  replaced 
by  a 2-tuple.  For  the  Bernolli  process  with  binomial  sampling. 


and  will 


V' 

Z ~ (r,  n) , where  r = / ,x.  is  a sufficient  statistic 

mmm *0  1 

therefore  replace  the  n - tuple  in  the  analysis.  Thus  equation 
(9)  may  be  written  as : 
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(10) 


£ (r|p) 


(n\  r . n-r 

a-p) 


r = 0 , 1 , 2 . . . .n 


If  we  put  a beta  prior  distribution  on  p,  the  posterier  distribution 
is : 


(H) 


(P|r)  = 


(n+A+B-1) ! 
(A+r-1) ! (n+B-r-1)! 


A+r-1 


(1-P) 


n+B-r-1 


The  problem  now  is  to  find  a function  0 defined  by  P = 0 (r)  which 
will  minimize  the  posterier  expected  risk  in  estimating  p.  The 
definition  of  expected  risk  is : 


(12) 


(0,  p)  f (p)  dp 


The  concept  of  correct  action  leads  to  the  definition  of  another 
term,  the  loss  function,  denoted  by 


L j^0  (r);  pj 

The  loss  function  is  intended  to  give  the  loss  which  is  incurred 
when  a certain  action  is  taken  when  a certain  state  of  nature 

prevails.  Since  the  interest  is  in  the  estimate  of  p being  close 
to  p,  there  difference  should  be  small.  For  this  purpose  we  use 
the  squared  error  loss  function  which  is : 


L [0  (r);  p]  = (p-p)2 


The  posterier  risk  then  can  be  written  as : 


ITT 


/ ^ (n+A+B- 1) ! 

C13)  V Cp;r)  = (£7-1)1  (n+B-r-1) ! 


Op) 


n+B-r-1 


dp 


/ 


/ .2  A+r-1 

CP-P)  P 


_ "2  0 " (A+r)  + (A+r)  (A+r+1) 

P (n+A+B)  (n+A+B)  (n+A+B+1) 


The  value  of  p as  a function  of  the  sample  which  will  minimize  the 
posterier  risk  is  then  obtained  by  taking  the  derivative  of  v (p;r) 

A 

with  respect  to  p and  setting  the  result  equal  to  zero;  that  is: 


a v (p;r)  = 

A 

a p 


2 p - 2 


A+r 

n+A+B 


= 0 


or 


(14)  r = p (n+A+B)  - A 


Thus  an  optimum  decision  rule  has  been  obtained  for  finding  the  value 

of  c in  terms  of  p.  The  criteria  for  finding  the  number  of  allowable 
defectives  in  a sample  of  n can  be  given  as 


Reject  if:  c^  > p^  (n+A+B)  - A 

Accept  if:  c^  <_  p-^  (n+A+B)  - A 

This  criteria  can  readily  be  extended  to  more  than  two  categories 

by  specifying  the  allowable  lot  fraction  defective  P,  for  each 
of  k categories . 

If  another  lot  is  to  be  tested,  A+r  and  n+B-r  can  be  used 
as  parameters  for  a new  prior  distribution.  It  may  be  noted  that 
after  a number  of  lots  have  been  tested,  the  terminal  analysis 
tends  to  become  less  sensitive  to  the  parameters  of  the  initial 
prior  distribution  and  more  dependent  on  the  accumulated  test 
experience . 
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6,  CONCLUDING  REMARKS.  The  decision  theory  approach  applied 
to  the  sampling  situation  in  this  paper  is  productive  since  a small 
amount  of  experimental  data  is  combined  in  a rationale  manner  to 
make  decisions  among  alternative  courses  of  action.  It  is  assumed 
that  prior  test  experience  is  available  so  that  the  prior  distribution 
may  be  determined  since  the  entire  structure  of  prior  convictions  is 
expressed  by  the  beta  distribution.  Also,  the  method  considered  does 
not  depend  on  the  specification  of  consumers  on  producers  risk  but 
only  requires  the  experimenter  to  answer  a non-statistical  question 
about  good  or  bad  quality.  That  is,  what  percent  defective  is  accept- 
able for  the  lots  under  test?  In  most  practical  situations,  this 
question  can  easily  be  answered. 
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PSEUDO-BAYESIAN  INTERVALS  FOR  RELIABILITY  OF 
A SERIES  SYSTEM  GIVEN  WEIBULL  COMPONENT  DATA 
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ABSTRACT,  A pseudo-Bayesian  solution  for  confidence  intervals  on 
average  reliability  of  a series-system  composed  of  Weibull  components 
has  been  formulated.  The  term  pseudo-Bayesian  is  used  since  the  goal 
is  to  choose  priors  that  lead  to  classical  limits  and  not  the  usual 
Bayesian  limits.  Uniform  priors  are  assumed  for  population  param- 
eters to  approximate  complete  prior  ignorance.  The  distribution 
assumed  for  component  interarrival  failure  times  is  the  2-parameter 
Weibull  with  both  parameters  unknown. 

In  the  solution  derived,  an  approximation  is  used  to  compute 
average  system  reliability  from  average  component  reliabilities.  A 
normal  distribution  is  then  assumed  for  the  log  of  system  reliability 
with  mean  and  variance  being  computed  from  the  posterior  means  and 
variances  of  the  individual  component  log-reliabilities.  The  bias  in 
the  mean  log-reliability  was  also  investigated  and  an  unbiasing 
factor  can  be  used  to  reduce  the  potentially  large  errors  in  system 
reliability  resulting  from  the  accumulation  of  biases  in  the  component 
means. 

Monte  Carlo  trials  were  conducted  to  determine  frequency 
exactness  of  the  derived  intervals  for  particular  cases.  Near 
exactness  was  observed  for  a number  of  cases  depending  on  Weibull 
shape  parameters,  true  component  reliabilities  and  sample  sizes. 


NOTATION 


f(t) 

F(t) 

F(t) 

w-) 

h(t) 


"c 

nf 

nm 


pdf  of  interarrival  times  of  failures; 
cdf  corresponding  to  f ( t ) ; 

1 - F(t) 

cdf  of  average  reliability  Ra; 

renewal  rate;  the  unconditional  pdf  of  component 
failure  and  subsequent  renewal; 
number  of  components  in  system; 

number  of  component  failures; 

number  of  missions  over  system  life; 
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R(t,i) 
R 


l(t,T) 

a(t) 


R 

u ' 

Rja(T) 
Rs(t,i) 
Rsa(x ) 


xf 

a 

3 

T 


reliability  at  time  t for  an  interval  x; 
reliability  of  the  jth  component; 

average  reliability  over  system  life  for  mission 
interval  t; 

average  reliability  of  the  jth  component; 
system  reliability  at  time  t for  an  interval  t; 
average  system  reliability; 

system  or  component  age; 
starting  time  for  the  ith  mission; 

sample  outcome  including  both  failure  and  censoring 
times; 

component  failure  time; 

Wei  bull  scale  parameter; 

Wei  bull  shape  parameter;  and 

mission  length  for  which  reliability  is  required. 


1.  INTRODUCTION.  The  general  problem  is  to  determine  the 
reliability  of  a series  system  from  component  test  results  where  the 
term  "series"  implies  that  a failure  of  any  component  in  the  system 
results  in  system  failure.  The  interarrival  failure  times  of  each 
component  are  assumed  to  follow  the  2-parameter  Wei  bull  distribution 
with  both  parameters  unknown.  In  addition,  the  assumption  of  ideal 
repair  is  made  wherein  a component  is  instantaneously  renewed  with  a 
like  new  component  whenever  it  fails  during  system  operation. 
Finally,  a fixed  number  of  failures  with  associated  failure  times 
are  assumed  given  for  each  component. 

Consider  first  a single  component  in  the  system.  The  failure 
times  for  a component  subject  to  ideal  repair  form  a renewal 
process.  The  theory  for  renewal  processes  and  interval -rel iabil ity 
are  well  covered  in  the  literature;  so  only  the  final  results  are 
summarized  here  [1-5].  The  failure  times  of  components  within  a 
system  are  not  known  in  advance  and  are  treated  probabil istically  by 
introducing  the  renewal  rate  (unconditional  failure  rate)  h ( t ) over 
the  population  of  all  systems.  The  renewal  rate  in  this  case  is 
distinguished  from  the  hazard  or  conditional  failure  rate  which 
describes  failure  of  a non-repairable  item.  The  renewal  rate  is  a 
function  of  the  underlying  failure  distribution  [2,  3]: 


h(t)  ~ f(t)  + / f (t-x)h(x)dx, 


(i) 


Interval  or  mission  reliability  at  system  time  t for  mission  length 
x can  be  determined  from  the  renewal  rate  [4]: 


(2) 


t _ 

R( t ,T ) = F ( t+x ) + / F(t+x-x)h(x)dx 

o 


in  which 

F(t)  = 1 - exp  (-at^ ) 


(3) 


for  the  Wei  bull  distribution. 


Since  interval -rel iabil ity  given  by  (2)  is  transient,  there  is 
some  motivation  to  define  a single  reliability  index  that  would 
characterize  a component  in  a system  throughout  system  life.  For 
this,  one  can  define  the  worst  mission  reliability,  the  asymptotic 
reliability  or,  as  is  done  in  this  paper,  the  arithmetic  average 
reliability  for  some  fixed  system  life  given  in  number  of  missions: 


nr 


'm 

■m  w 


yo-i -i 


(4) 


in  which  t^  - starting  time  for  the  ith  mission. 

Average  component  reliability,  as  defined  by  (4),  does  not 
readily  lend  itself  to  classical  confidencing  approaches  since  a 
statistic  for  reliability  could  not  be  found  which  depends  only 
on  the  true  reliability.  The  statistics  considered  depended  on  both 
of  the  unknown  Wei  bull  parameters.  This  is  in  contrast  to  reliabil- 
ity of  a non-repairable  component  in  which,  for  the  Weibull  example, 
the  distribution  of  the  maximum  likelihood  estimate  of  reliability 
depends  only  on  the  true  reliability  [6,  7], 


A Bayesian  approach  was  consequently  used  to  at  least  render 
this  problem  numerically  tractable.  The  goal  was  not  to  determine 
the  usual  Bayesian  limits,  in  which  prior  information  is  to  be  used, 
but  rather  to  choose  priors  which  gave  near  classical  frequency 
limits;  hence  the  term  pseudo-Bayesian. 


The  solution  for  the  single  Weibull  component  in  which  uniform 
priors  were  assumed  for  the  population  parameters  is  presented  in 
another  paper  [8].  The  final  result  for  the  posterior  cdf  for 
average  component  reliability  from  this  paper  is  given  by  the  follow- 
ing expression: 


FRa(z|X;t,nn])=K'1  / a(S ) (b(6 ) ) 1 P(nf+l,w(T,nm,z,B))dB  (5) 


-nf-l 


w 


-nf-l 

in  which  K = / a (a ) (b(3 ) ) dp 
o 

nf 

a(e)  S 8 f II  (xfi )&_1 
i=l 


b(6)  = l x/ 
j=l 

N = total  sample  size  including  both  failures  and  censor- 
ing times 

w(x,nm, z,8)  = R"1(i,nm,z,B)b(S) 

P(n;x)  = incomplete  gamma  function 
n-1 

- l-e"x  l xVi!  for  integer  n [9]. 
i=o 


Solution  of  (5)  for  given  z,  t,  nm  and  sample  outcome  X is  accom- 
plished by  numerical  quadrature.  Confidence  limits  on  component 
reliability  can  be  determined  from  (5)  by  computing  the  probability 
limit  z for  a given  probability  level. 

In  this  paper,  results  are  presented  for  a few  of  the  problems 
encountered  in  determining  system  reliability  from  the  component 
results  using  a similar  pseudo-Bayesian  approach  and  using  the 
derived  posterior  distribution  (5)  for  average  component  reliability. 


2.  NUMERICAL  COMPUTATION  OF  SYSTEM  RELIABILITY.  The  first 
problem  encountered  is  the  computation  of  average  system  reliabil  ity 
from  the  average  component  reliabilities.  The  reliability  of  a 
series  system  is  given  as 


Rs (t  ,T  ) 


Mc 

TT  Rj(t.t).  (6) 

j=1 


Average  system  reliability  can  be  defined  in  a similar  manner  as 
average  component  reliability: 


RS3(t) 


1 


'm 


Mm 

l 

1-1 


nc 

I I Rj  ( ti  *T ) 
jfil 
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(7) 


This  is  a difficult  equation  to  work  with  since  the  time  dependent 
component  reliabilities  are  required.  Ideally,  one  would  like  to 
express  average  system  reliability  in  terms  of  average  component 
reliabilities  so  that  use  can  be  made  of  the  posterior  distribu- 
tion given  by  (5). 

Three  approximations  to  average  system  reliability  were 
investigated: 

nc 

R$a(t)  = l-[  (l-Rja(T>)  (8a) 

j=l 

nc 

= exp  [-  l (1-Rja(t))]  (8b) 

j=1 


“ JJ  Rja^>  <8c> 

J=1 

In  the  first  two,  component  reliabilities  were  approximated  by 
exponential  forms  which  are  accurate  for  high  reliability  components. 
In  the  third,  the  product  and  summation  signs  for  system  reliability 
(7)  have  been  interchanged  and  the  definition  of  component  reliabil- 
ity (4)  was  used.  Equality  would  exist  in  this  third  case  if  geo- 
metric averages  had  been  considered  instead  of  arithmetic  averages. 
The  geometric  average  is  close  to  the  arithmetic  average  for  either 
high  reliability  components  or  if  there  is  a relatively  small  varia- 
tion in  reliability  as  a function  of  time.  Table  1 lists  some  of 
the  computations  performed  using  (8a),  (8b)  and  (8c).  Based  on  these 
and  other  computations,  it  was  concluded  that  (8c)  represents  an 
adequate  approximation  to  system  reliability  over  a wide  range  of 
reliability  levels.  It  also  gives  a somewhat  conservative  result  in 
that  a lower  than  true  reliability  is  generally  computed. 


3.  BAYESIAN  SOLUTION  FOR  SYSTEM  RELIABILITY.  The  next  step  is 
to  formulate  a feayesi an  solution  Tor  average  system  reliability. 
Taking  logs  of  (8c^  gives 

it)  Rsa (t ) = l in  Rja(x).  (9) 

j=l 

This  equation,  in  a Bayesian  sense,  represents  m Rsa  as  a random 
variable  which  is  equal  to  the  sum  of  the  random  variables  Jtn  R.  . 
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Using  the  Central  Limit  Theorem,  the  posterior  distribution  of  An 
Re,  can  be  assumed  to  asymptotically  approach  the  normal  distribu- 

tion.  The  mean  and  variance  of  An  Rsa  required  in  the  normal  dis- 
tribution can  be  derived  from  the  means  and  variances  of  An  R,_: 

nc  ja 

E(*n  Rsa)  = l E(tn  Rja)  (10a) 

j = 1 

and  nc 

Var  (An  Rc, ) * l Var  (An  (10b) 

sa  u Ja 

j^l 

in  which  statistical  independence  is  assumed  for  the  random  variables 
An  Rja.  It  remains  then  to  determine  E(An  Rja)  and  Var  (An  Rja ) 

from  the  posterior  distribution  of  Rja.  These  are  obtained  from  the 

following  relations: 

1 

E(j>n  R-a)  = - / (Fr  (z)/z)dz  (11a) 

0 J & 

O 1 

Um  Rja)  = -2  / (Fr.  (z)(>nz/z)dz  (lib) 

o 


Var(*n  R.  ) = E(*-n2  R.  ) - EZ(£n  R.  ) (11c) 

ja  ja  ja 

in  which  Fr  (z)  is  given  by  (5). 
ja 

Using  the  normal  distribution  assumption  for  An  R$a,  prob- 
ability limits  on  can  then  be  determined  using  a standard 

sa 

normal  table. 


4.  BIAS  OF  THE  BAYESIAN  ESTIMATES.  From  previous  work  and 
from  work  on  similar  approaches  presented  in  the  literature,  the  final 
solution  for  system  confidence  limits  can  be  sensitive  to  the  bias 
of  the  estimators  used  for  component  reliability.  That  is,  if  the 
component  reliability  estimates  are  biased,  then  a function  of  these 
such  as  system  reliability  can  become  highly  biased.  The  statistical 
bias  of  component  log  reliabilities  was  consequently  studied  to 
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determine  potential  bias  of  the  system  reliability  estimates. 

Table  2 summarizes  some  of  a number  of  results  obtained  for  the 

true  mean  and  for  Bayesian  mean  of  £n  R.  derived  from  Monte  Carlo 

ja 

simulation.  Generally,  the  results  indicate  that  the  mean  of  in 
Rja  is  biased  on  the  conservative  side  of  true  reliability.  Unbias- 
ing factors  were  studied  which  are  analogous  to  the  exponential  case 
where  one  failure  is  subtracted  from  the  total  number  of  failures  to 
yield  an  unbiased  estimate  of  mean  failure  rate  [10].  Unbiasing 
factors  obtained  by  subtracting  0.5  to  1.0  failures  gave  the  best 
overall  results  for  Wei  bull  shape  parameter  in  the  range  of  2 to  6 
as  indicated,  for  example,  in  Table  2. 


5.  FREQUENCY  INTERPRETATION  OF  THE  BAYESIAN  INTERVALS.  The 
final  question  regarding  the  Bayesian  limits  is  whether  or  not  there 
is  a frequency  interpretation  of  the  resulting  intervals  with  and 
without  an  unbiasing  factor.  In  order  to  check  this,  a number  of 
Monte  Carlo  simulations  were  conducted  using  the  previously  described 
confidencing  procedure.  Monte  Carlo  is  not  used  here  to  derive  the 
confidence  intervals  but  rather  to  check  for  exactness. 

Various  systems  of  3 and  6 components  were  assumed.  The  shape 
parameter,  true  reliability  and  number  of  failures  for  each  component 
were  fixed  at  different  assumed  values.  Test  samples  were  then 
artificially  generated  from  random  numbers  using  the  assumed  parameters. 
Exactness  was  then  checked  for  the  generated  samples.  For  these  trials 
the  system  life  was  150  missions  with  mission  time  r being  equal  to 
1.0. 


Table  3 lists  some  of  the  results  of  the  Monte  Carlo  trials.  In 
this  table  the  Kolmogorov-Smirnov  rejection  error  is  presented  [11]. 

This  error  represents  significance  level  or  risk  in  rejecting  the 
hypothesis  that  the  confidence  intervals  are  exact  at  all  confidence 
levels  when  in  actuality  the  hypothesis  is  true.  From  the  results 
given  in  Table  3 it  can  be  seen  that  when  no  unbiasing  factor  is 
used,  the  resulting  confidence  intervals  are  not  exact  at  all  confidence 
levels.  Using  an  unbiasing  factor  of  (n^-0. 5)/n^  gave  the  best  overall 

results  except  for  the  case  of  6^1.0. 

In  addition  to  the  K-S  statistic  given  in  Table  3,  the  relative 
distribution  of  the  confidence  limit  was  also  generated  to  determine 
exactness  for  the  lower  confidenced  reliability  in  the  range  of  90  to 
9S%  confidence.  It  was  found  that  in  all  cases  considered  the  lower 
confidenced  reliability  is  conservative  when  no  unbiasing  factor  is 
used.  That  is,  the  proportion  of  the  time  that  the  true  reliability 
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TABLE  2 


SUMMARY  OF  COMPUTATIONS  TO  CHECK  BIAS 
OF  BAYESIAN  MEAN  OF  LOG  OF  RELIABILITY 


n 


f 


eu 


V1 


l( 


5 

-) 


True  Ra 

a 


H 

vS" 


10 

1.0 

,879 

.890 

.885 

.900 

2.0 

.897 

.907 

.902 

.900 

3.0 

.896 

.305 

.901 

.900 

1 0 

1.0 

.940 

.945 

.943 

.950 

2.0 

.947 

.953  I 

.950 

.950 

3.0 

.949 

.954  1 

.951 

,950 

1 

6.0 

.949 

.954 

.951 

.950 

10  ! 

1.0 

,989 

,9901 

.9895 

.990 

2.0 

.9891 

.9902 

.9896 

.990 

u = Mean  of  R from  Monte  Carlo  Simulation  (1000  trials) 

a 

rif  - Number  of  Failures 
$ = Wei  bull  Shape  Parameter 


TABLE  3 


hi 

■iW 


H 

o 


K-S  TEST  FOR  MONTE  CARLO  TRIALS  TO  CHECK  EXACTNESS 


K-S,  Rejection  Error* 

Run  No  Unbiasing  Unbiasing  Factor 


No. 

nc 

RJ 

e3 

nfj 

Factor 

- (nf-0.5)/nf 

I 

3 

.90, 

.90,  . 

90 

1, 

2, 

3 

20, 

20, 

20 

0.05 

>0.20 

2 

3 

.90, 

.90,  . 

90 

1, 

2, 

3 

10, 

10, 

10 

0.01 

0.10 

3 

3 

.90, 

.90,  . 

90 

1, 

1, 

1 

20, 

20, 

20 

0.01 

0.01 

4 

3 

.90, 

.90,  . 

90 

2, 

2, 

2 

10, 

10, 

10 

0.01 

0.15 

5 

3 

.90, 

.90,  . 

90 

3, 

3, 

3 

10, 

10, 

10 

0.01 

>0.20 

6 

6 

0.97, 

0.97, 

0.97, 

1, 

2, 

3, 

10, 

10, 

10, 

0.01 

>0.20 

0.97, 

0.97, 

0.97 

4, 

5, 

6 

10, 

10, 

10 

7 

6 

0.90, 

0.90, 

0.90, 

1, 

2, 

3, 

10, 

10, 

10, 

0.01 

>0.20 

0.90, 

0.90, 

0.90 

4, 

5, 

6 

10, 

10, 

10 

8 

6 

0,90, 

0.90, 

0,90, 

1, 

2, 

3, 

5, 

5, 

0.01 

>0.20 

0.90, 

0.90, 

0.90 

4, 

5, 

6 

5, 

5, 

5, 

9 

3 

0.95, 

0.95, 

0.95 

3, 

3, 

3 

10, 

10, 

10 

0.01 

0.15 

10 

3 

0.99, 

0.99, 

0.99 

2, 

2, 

2 

10, 

10, 

10 

0.01 

>0.20 

*Significance  or  risk  in  rejecting  hypothesis  that  confidence  intervals  are  exact  at  all  levels 

when  hypothesis  is  true. 


was  greater  than  the  lower  confidence  limit  was  at  least  equal  to 
the  confidence  level.  These  results  hold  for  Wei  bull  components  with 
shape  parameter  greater  than  1.0. 

It  is  also  of  interest  to  compare  the  pseudo-Bayesian  limits 
derived  for  Wei  bull  components  to  the  usual  method  used  for  determin- 
ing confidenced  reliability  from  component  results.  Generally,  the 
exponential  assumption  is  made  for  failure  time  distribution  regard- 
less  what  the  true  underlying  distribution  may  be.  The  exponential 
assumption  yields  a constant  failure  rate  (constant  reliability) 
for  components  and  hence  for  the  entire  system.  This  assumption  is 
usually  made  since  confidence  intervals  can  often  be  derived  clas- 
sically [10].  Table  4 lists  some  results  of  the  average  lower  90% 
confidence  limit  on  reliability  for  the  pseudo-Bayesian  Wei  bull  and 
the  classical  exponential  methods.  From  these  results  it  can  be 
seen  that  the  Bayesian  limits,  although  previously  shown  to  be  con- 
servative, are  not  as  conservative  as  the  exponential  limits.  The 
degree  of  difference  depends  on  true  reliability  and  shape  parameter. 


6.  CONCLUSIONS.  Although  numerically  tedious,  the  pseudo- 
Bayes  IarTmHHoBoT^wnfi  dene  ing  system  reliability  for  Weibull 
components  described  in  this  paper  appears  to  be  a sound  approach 
to  a problem  which  generally  has  no  other  solution.  Two  conclu- 
sions that  can  be  made  based  on  the  results  of  this  study  are: 

a.  The  approximation  m Rsa  = £ £n  Rja  f°r  system  reliability 

with  no  unbiasing  factor  gives  a conservative  lower  con- 
fidence limit  for  all  cases  considered  but  not  as  conserva- 
tive as  assuming  the  exponential  distribution. 

b.  The  unbiasing  factor  (n^-0.5)/nf  to  (nf-1.0)/nf  gives  the 

best  overall  results  for  exact  confidence  limits  at  all 
confidence  levels  for  s>1.0. 
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TABLE  4 


AVERAGE  LOWER  90%  CONFIDENCE  LIMIT  FOR  BAYES IAN- 
WEIBULL  AND  CLASSICAL  EXPONENTIAL  METHODS 


Trial 

No. 

No.  of 
Comp. 

True 
Comp. 
Rel . 

Wei  bull 
Shape 
Parameter 

No.  of 
Failures 
Per  Comp 

True 

System 

Rel. 

Average  Lower  90%  Confidence 
Limit  on  System  Reliability^ 

Wei  bull ^ 

Exponential ^ 

1 

3 

.995 

6.0 

10 

.985 

.982 

.965 

2 

3 

.995 

3.0 

10 

| .985 

.979 

.967 

0 

r- 

. aso 

C*\j 

10 

1 .985 

.978 

.968 

i 

! oqc: 

* -'w 

5.  G 

! ,985 

i 

. 979 

i 

.965 

5 

i 

3 

! .995 

3.0 

5 

.985 

.981  ! 

i 

.968 

6 

3 

,995 

2.0 

5 

.985 

.970 

.965 

7 

6 

.995  ! 

2.0 

10 

.970 

.959 

.941 

8 

10 

.995 

2.0 

10 

.951 

.940 

.910 

9 

3 

.990 

2.0 

10 

.970 

.963 

.956 

10 

3 

.950 

2.0 

10 

.857 

.831 

.829 

(1)  Average  of  10  Monte  Carlo  trials. 

(2)  Pseudo-Bayesian  limits  described  in  this  paper. 

(3)  Mann-Grubbs  frequency  limits  [10]. 
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ABOTjCl OT,  The  paper  deals  with  simulating  + 'nr  reliability  predictions 
ol  heavy  Lift  Helicopter  (dL.l)  Fly  -by  -'lire,  system*  Acceptance  decisions 
are  dependent  on  the  prior  distributions  (based  on  available  subjective 
judgments  and  experimental  information)  of  the  equipment  del;]. abilities 
which  constitute  the  "states  of  nature"  (Ai). 

The  prior  probabilities  F(A_- ) associated  with  the  states  of  nature 
(reliabilities)  are  - deterra  ned  from  test  data , historical  system  reliablH  ■ 
data  predj  oti  ons , and  i rtf  erences  from  experiences  with  sit'd  lar  equipments. 


The  A.:  * s are  arbitrary  discrete  values  near  the  midpoints  of 
various  relish-'  i.ity  bands.  These  arc  chosen  to  cover  the  reliability 
scale  from  0 to  1.  A0,  for  example , is  the  event  witidn  cell  No.  ? with 

its  associated  reliability  of  0,99?  999  99:  P(A^)  - • 19(d),  'The  cell 
values  (midpoints) are  chosen  to  be  closer  to  one  another  at  the  high*  r 
values  of  reliabilities  and  spread  apart  os  the  reliabi Titles  decrease. 


Tills  is  done  with  the  objective  of  obtaini  up  a sharp  d ec erinii ia ti on  of 
reliability  values  at  the  required  gr,  spectrum.  This  ore  rvf  a-  wi  Li. 
enable  one  to  investigate  the  associated  probabilities  at  ti >de  er 
reli.abili l.v  regions,  dee  Table  lend.  F;lf 'ure  I. 

He  liability'  tests  were  performed,  arid  Bayesian  algorithms  were 
used  to  update  the  priors  based  on  the  test  data.  This  results  in  a 
posterior  distribution  which  is  used  as  a new  prior  for  the  sent  phase 
of  testing . 


Various  initial  priors  were  applied  to  di r fn.rc-nt  sets  of  test  data 
in  order  to  observe  how  sensitive  the  mi  ors  arc , after  Bay  e si  ar.  updst:. 
in  af.fecti  ng  the  re  an  ’.taut  posterior  clistri  Imti  ons,  Jharts  and  grapes 
showing  these  results  are  o resented . 
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Table  I.  Discrete  Reliability  Cell  Formulation 

Cell  Reliability  Reliability  Rands 

Values 


1 

0.999 

999 

9 ?5 

0 . 999 

999 

992 

5 » 

1.0 

0 

c. 

0,999 

999 

99 

0.999 

999 

975 

- 

C .999 

000 

^ y ^ 

3 

0.999 

999 

95 

0.999 

}J9 

925 

- 

O.W 

999 

*h 

0.999 

999 

9 

0.999 

009 

75 

- 

0.999 

999 

LJ 

ry 

0.999 

999 

5 

0,999 

999 

25 

- 

0.999 

999 

6 

0.999 

999 

0,999 

997 

5 

- 

0.999 

999 

i 

0.999 

995 

0.999 

992 

- 

0.999 

99  7 

d 

0.999 

99 

0. 999 

975 

- 

0.999 

992 

9 

0. 999 

9 

0.999 

75 

- 

0 . 999 

975 

.10 

0.999 

0.997 

- 

0. 559 

75 

11 

0,99 

0.975 

- 

0.997 

6 

l2 

0.9 

0.85 

- 

0,975 

,n 

0.8 

0.75 

- 

0,8.5  ■ 

Ht 

0.7 

0.65 

- 

0.75 

lb 

0.6 

0. 5 b 

- 

0,65 

16 

0.5 

o.l  1.5 

- 

0.55 

I? 

0 , ll 

0,35 

- 

O.ljb 

lo 

0.  3 

o.?5 

- 

0 . 35 

1) 

0,2 

0.15 

- 

0.25 

20 

0.1 

0.05 

- 

0.1 5 

992  9 


^Thic  is  the  reliability  we  would  like  to  obtain 


! .0  BAOiCOROUND*  Early  in  the  i nvc * s ti. Rations  of  the  heavy  L:i  ft 
Ho l:i  copter * s ( UL il)  development , it  was  recogni  zed  t.liat  cortai n areas  or' 
its  development  would  be  s technological  cliallenpe,  The  area  of  flight 
controls  stimulates  avionics  interest  due  to  the  needs  of  container  oary 
off  shore  loading  9 Fly  »3y  -Wire  (FBW)  primary  control,  precis  lor  hovering, 
and  a ling  load  stabilization, 

A ms/, or  technological  breakthrough  in  the  KLII  is  the  FBW  urinary 
control . In  this  approach  the  conventional  mechanical  urinary  flight 
enr.trd  mechanical  linkage  is  eliminated  and  the  pilot's  stick  commands 
t hr  control  surface  actuator  or  swashplate  through  electrical  signal 
wvrrs,  done  areas  of  primary  concern.!  in  the  use  of  a FBW  system  are : 
tl'ighl-  safety  reliabi  Li  ty  denonstration,  mission  reliabj.  11  tv  test  demons4 
and  nr eleai'  vu Lne.rabi  l.i  ty.  The  FBW  system  is  triple  redtindayi  wit  i a 
’Bright  safety  reliabi  Vi  ty  goal  of  O.opo  999  gyp  for  +wo  }lour5  0f  fynyy 
(or  a liTBl  ef  approximately.  10  ,000 y.XV  > hours), 

?-*'• ; JhTiMJBOTIOIJ.  The  objective  of  l.fris  paper  Is  lo  f i ; \, ' a 
technique  to  test  demonstrate  the  heavy  Lift,  ileli  rye ,!;r  (iOfj)  geipb; 
redundant  }"!y  -Uy-Jivn  (FBW)  system’s  flight  safety  reliability  goal  of 
0*999  ')■■)(■>  uyjj  t'0j.  two  hours . It  investigates  by  simulating  the  roll  obi!  1. 
predictions  of  the  HT.il  FBW  system. 


Id.na.LIyj  the  purpose  of  a reliability  demonstration  technique  is  to 
establish,  in  the  shortest  possible  test  time  and  at  the  minimum  cost, 
whether  tin's  high  reliability  goal  can  be  met.  The  FBW  reliability 
demonstration  testing  environment  consists  of; 

* Hi  gh  Reliability  Requirements 

Limited  Funds 

GUort  Test  Times  Available 

Low  Test  Risks  (which  requires  long  test  times) 


In  other  words , it  is  desirable  to  have  low  producers ’ and  consumers  1 
risks  winch  results  in  .long  test  times  due  to  the  high  reliability 
requirements.  Generally,  the  accuracy  of  the  demonstration  reliability 
tests  and  the  measure  of  test  confidence  increases  as  the  number  of  observed 
failures  .increases . However,  high  reliability  requirements  mean  long 
tines  to  (observed)  failures . This  environment,  creates  a cost/ time 
pro  cion  wcich  is  apparently  unsolve Lie  by  traditional  Classical  methods. 

This  is  exactly  the  dilMM  involved  in  testing  the  highly  reliable 
and  '.-pa?';ciYc  HLH  FBW  system. 


when  one  applies  tun  traditional.  Classical  method  to  reliability 
'•V\;ouil ration  tests  the  following  cue rnotcri sties  emerge; ; 

T n.  technique  is  easy  to  use  cm  1 to  understand 

It  Is  assumed  tliat  the  d,  riroe  reliabili  ty  R is  Pino  1 (R  is 
actually  a random  variable) 


Previous  (or  prior)  failure  information  is  ignores  isuch  as 
laboratory  design  and  development  tests , initial  buyer  acceptance 
tests , etc . ) 


* It  requires  long  test  tines  for  equipments  of  high  reliability 


After  investigations,  the  conclusion  was  reached  that  if  is  not 
practical  to  use  traditional  Classical  tcclmiques  to  test  demonstrate 
systems  of  0*9C99  999 or  higher  with  a reasonable  degree  'of  confidence. 

As  an  alternative,  Bayesian  statistical  techniques  arc  considered . 

In  this  methodology  it  is  essential  that  a prior  distribution  ir  used  and 
the  reliability  parameter  R he  considered  as  a random  variable . 
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3.0  BAYdb  TdhURhjR  Bn  yes  Theorem  is  •essentially  a simple  relation 
between  j i ro Bab i li t ies  of  the  occurrences  of  two  different  events.  It 
will  be  applied  in  tide  paper'  to  a discrete  reliability  cell  formulation. 

The  basic  expression  which  describes  the  Bayes  relations! dp  between 
two  events  is: 


P(A,/3)  - ' (A0  ?Ci/'0 


P(ii) 


w 


n 

here  P(B)  = V P(Aj  ? (B/Aj 

i=l 


and  tneV'is  to  be  read  ''given  that". 


Tills  relationship  ih  words  s 


New  Probabil  li  ty 

Prior  Probab.i  1:1.  ty 

v oJr  > c e mini ; s \ > e c i fi  ed 

- 

c one e rn ing  spec if i e d 

r elii.a  bill  ty  hypothec  is 

reliabi li ty  lypothesis 

i — 

r-1 

Q 

rn 

U 

h 

\ 

Probabil:.  ty  of 
Observed  Test 
outcome  under  the 
Reliability  Nypothe si s 


Total  Probabil 1 ty  cf 
Observed  test  c‘itcvi,t..j 
uruer  all  apecified 

Reliabili  ty  itypotheses 


The  terms  of  the  relationship  are  defined  as  follows: 

Ay:  States  of  nature;  or  the  event  of  having  cell  number  'i'1  with  its 

associated  reliability , 


B:  test  event;  or  evidence  bearing  upon  either  the  success  or  failure 

of  an  observed  reliability  test  (that  is,  test  result,  which  has  a bearing 
upon  the  credibility  of  the  Aj_  events). 

p(A.  ) : Reliability  associated  with  cell  number  ,fi,f,  given  by  prior  distribution 

assignments, before  test  evidence  B becomes  available. 


"!  >Hfsc  of  bay  os  ' Theorem  In  Its  discrete  Formulation  lor  Reliability 
ii-ti  mat  ion  Purposes ",  W.J.  Mac  Far  Land,  from  Transactions  of  the  7 th 
it  liability"  and  "llaiiit  a i,  nabi  li  ty  Conference,  July  1 9(6,  pages  35'? -365, 

1:  6b  Annual  Assurance  Sciences. 
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P(B/JLj ) ; The  nrobabili  ty  of  tiio  observed  outcome  B assuming 

the  occurrence  of  event  A^.  Q'The  probability  that  the  test 

resulted  in  a success  (the  equipment  w< rf  ed  satisfactorily  at  the  end 
of  a fixed  time  interval)- given  that  the  associated  cell  ib  r»I1 ability 
is,  say,  0.90 -is  0.90,  Also,  the  probabi  1.1  ty  of  an  observed  test 
failure  B-  given  that  the  associated,  eel.],.  Aj  reliability  is,  say,  0.90- 
is  obviously  0.10]. 

P ( B ) : Tiio  probability  of  the  observed  event  B evaluated  across 

the  entire  wei  ghted  ensemble  of  events  iU  . 

P(A^/B):  The  posterior  or  modified  (new)  probability,  or  trie 

probability  assigned  to  events  Ay  as  a result  of  the  new  test  evidence 
B by  the  use  of  Bayes1  technique.  Thus,  Bayes1  Technique  provides  the 
basis  to  recompute  P(A. ) based  on  additional  new  test  evidence  B. 

h .0  BAYBBIAh  DISCRETE  UPBATI’iG.  Table  L shows  a discrete  reliability 
cell  formulation  consisting  of  21  cells  and  21  reliability  ranges  ( the 
number  of  chosen  cells  is  arbitrary; . The  Aj  *s  arc  arbitrary/  discrete 
values  near  the  midpoints  of  the  various  reliability  bands  or  ranges. 

These  bands  arc  chosen  to  cover  the  reliability  scale  from  0 to  l.  The 
cell  reliability  values  (midpoints) , the  A^’s^are  selected  close  to 
one  another  at  the  desired  reliability  of  interest; .and  spread  apart  as  they 
diverge  from  the  desired  reliability.  This  is  done  with  the  objective 
of  obtaining  a sharp  discrimination  of  reliability  values  for  the  required 
high  spectrum.  This  procedure  enables  one  to  investigate  the  associated 
probabil^fies  in  the  higher  reliability  regions. 


Bayesian  acceptance  decisions  are  dependent  on  the  prior  distributions 
(based  on  available  subjective  judgments  and  experimental  information)  of 
the  system  reliabilities  which  constitute  the  '’states  of  nature !1  (A^) . 


The  next  step  is  to  establish  from  Historical  and  empirical  data  a 
prior  reliability  distribution.  One  must  determine  from  existing  historical 


and  empirical  failure  data  the  probability  values,  P(Aj),  for  each  cell 
reliability  value  Aj . 


1.0  UBTIiUMPiATION  OF  PKIOit  PROBABILITIES.  For  the  equipment  in 
question,  knowledgoable  individuals  such  as  the  component  manufacturers, 
the  design  engineers , reliability  experts,  and  other  responsible  individuals 
should  be  gathered  to  determine  a suitable  prior  distribution.  As  a 
hypothetical  example,  a component  manufacturer  is  asked  to  estimate  the 
percent  of  time,  a thousand  components  (in  the  equipment  in  question)  can 
be  expected  to  function  successfully  for  two  hours  within  the- cell  number  1 
reliability  band  (0.999  999  9929  to  1*0),  The  component  manufacturer  may 
indicate  one  percent,  i.e,  )P(A1)  = 0,01  based  upon  his  available  historical 
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TABLE  ( I ),  -DISCRETE  RELIABILITY  CELL  FORMULATION, 


( 

Cell 

No. 

STATES  OF  NATURE  ) 

A « Cell  Reliability 
Value 

■ Reliability 
Bands 

1 

0.999  999  995 

0.999  999  992  5 - 1.0 

2 

0.999  999  99 

0.999  999  975  - 0.999  999  992  5 

3 

0.999  999  95 

0.999  999  925  - 0.999  999  975 

*4 

0.999  999  9 

0.999  999  75  - 0.999  999  925 

5 

0,999  999  5 

0.999  999  25  - 0.999  999  75 

6 

0,999  999 

0.999  997  5 - 0.999  999  25 

7 

0,999  995 

0.999  992  5 - 0.999  997  5 

8 

0.999  99 

0.999  975  - 0.999  992  5 

9 

0.999  9 

0.999  75  - 0.999  975 

10 

0.999 

0.997  5 - 0.999  75 

11 

0.99 

0.975  - 0.997  5 

12 

0.9 

0.85  - 0.975 

13 

0.8 

0.75  - 0.85 

14 

0.7 

0.65  - 0.75 

15 

0.6 

0.55  - 0.65 

16 

0.5 

0.45  - 0.55 

17 

0.4 

0.35  - 0.45 

18 

0.3 

0.25  - 0.35 

19 

0.2 

0.15  - 0.25 

20 

o;i 

0.05  - 0.15 

21 

° 

0 - 0.05 
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-a'...'  a;"  bV/?9  9?9  r'  3 - the  most  u‘obe“ fiat  Is , P<A/)  - • y h?!).  ' 

. /a '.no  o t the  Av* rs fy  l5(Aj.)  ce/  jmr  r-.ov;  j ’orris  an  estimated  /rehab: 
sis ' r:i but:  or; , called  tne  prior  ddstrl/butioii  Pr;(A-i),  arid  is  * d; 


r ..  ..o;  ’■■an  ton  ;i r;  ..;  .1 . -urr 


‘an  show  a a prior  dis  i 


avy-rarcs.  deferring  to  Table  (2) , each  value  of  this  d:i  attribution  of 


averages  1 s incorporated 
(re Liabilj ty)  value  (Aj ) 
wit 't  cell  Ay  (0.990  pro) 

abscissa  in  figure  1. 


as  a prohibit ' ty  of  the  corresponding  ce  3.1 
. For  example . such  a probaM  U.tr  associated 
is  O.lib  in  Tabic  (2)  winch,  slotted  as  Vie 


The  cell  reliability  values  A?  ar-.„  drawn  cu  V;.o  aictoprav  r 
they  art:  not,  proportional  in  width  to  t'x;  anount  "/  probe  billy.  •: 
roliabi  Li  ty  va  lue  contained.  However,  in  this  paper,  each  Aj  is 

drawn  to  a uniform  scale  width  l. 


Using  the  his  bo pram  of  figure  1 as  our  prior , let  us  consiuer -some 
oxa.rplec  showing;  the  results  of'  the  Bayesian  updating  os Lcnlati  sr’s  upon 
this  prior  distribution.  Let  us  call  tills  prior  di  a *.:M<  ration  ’ it: 

binomial  type  average  prior* 


oU  BPLCIPIC  APPLICATION  OF  Ti,  .iJlhiJllPhi  BAifbJIAli  TLOLPIQUi.  TV 


PAiiTl.  01/ A I hsblADIljITT  T OLTd.  A test  is  performed  under  simulated 
environment;!  I conditions  which  are  expected  to  be  encountered  In  nature. 
1 or  example , a pi  eor  oh  Avioni.cs  equipment  ear:  Lx;  tested  during  actual 
.helicopter  tost  flip1! to  (and  ground  tests) , The  test  measurements  are 
to  record  the  failure  times  which  a. Lou  Id  indicate  whether  or  not  tiir 
oqui  ; merit  'T»  \ !.s  at.  t hr  cud  of,  say,  every  two  hours . 
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0.999  0.01  0.04  0.03  0.03  0.0275 


CELL  RELIABILH 


LXAhPLL  1.  T 
a VAlTmTnj^i  Fv 


first  test,  at 
• 3). 


trie 


end  of  two  hours,  resulted  in 


pAI FETAL  UPDATIiK 

based  upon  the  one:  empirical 


JOMPUTATIQNE.  He 
observe  t ion 


desire  to 
that  the  . 


update  each  P(A*/B) 
irst  test  resulted,  in 
^ / , . v*  wish  .x  \3.u  leua  uaow  i - 1 to  21; , is  the  set 
of  posterior  probabilities  of  , given  B.  These  can  be  used  as  the 

^ testing.  As  an  illustration,  the 


fai  lure . 


The  P(A,/B),  for  each  i (in  this  case 


new  priors  for  the  next  phase  o: 
calculations  for  or 11  values  Aq,  and  Aiq  are  given  below.  Tab! 


shows  the 
oxarple . 


result 


‘,S  0,J 


all  the  Bayesian  cell  value  calculations  for  this 


FOR  Ilf  Lb  VALUE  A; 

From  Table ( l)  Aq  = 0.999  999  ‘?91> , From  Table (2)  P(A].)  = 0.00325, 

we  want  to  compute  ?(A^/B). 

Here  A is  the  event  of  having  cell  number  1 with  its  associated 
reli ability^of  0.9 99  999  999,  P(A-j ) is  the  prior  probability  assigned  to 
cell  number  1 before  test  evidence  B is  available.  In  other  words,  it, 
is  the  prior  probabili ty  that  the  equipment  reli ability  falls  within  a 
band  of  reliability  (in  this  case  0.999  999  992  5 to  1.0)  centered 
approximately  about  0*999  99 9 995* 

Apply id-f  Bayes’  Theorem  we  get, 

PU-j/B)  - P(Ai)  P(B/Ai) 

P ( B ) 


ou+. 

rel 


otlier  words,  the  best  estimate  (based  on  th 
of  the  equl  pment  reli ability  falling  within  the  A]  He* 


The  Bayesian  Oalcnlations  are : 
r(A.j/Il)  - P(Aq)  P(  l^/An) 

p(b)  1 

n“2l 

F(B)  - JIT  i5(B/Ai)  P(/q)  (from  Table  3) 
x »1 

f(B)  - 0,()07l.|833 


: tv) 
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observed  fee 
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TABLE  ( 3 ).  BAYESIAN  UPDATING  CALCULATIONS  IF  FIRST  TEST  RESULTS  IN  FAILURE. 


Cell  Values, 

P (Ai) 

P (B/Ai) * 

P(B/Ai)  P (Ai) 

. _ .... 

P(Ai/B)„  P(M)P(B/A1) 
P (B) 

Ax  » 0.999  999  995 

0.00325 

0.000  000  005 

1.625  X 10"11 

0.000  00 

A 2 - 0.999  999  99 

0.0130 

0.000  000  01 

1.30  X 10'11 

0.000  00 

A3  « 0.999  999  95 

0.050 

0.000  000  05 

25  X 10“10 

0.000  00 

A4  - 0.999  999  9 * 

0.235 

0.000  000  1 

0.000  000  023  5 

0.000  00 

A5  - 0.999  999  5 

0.2425 

0.000  000  5 

0.000  000  121  25 

0.000  02 

Ag  « 0.999  999 

0.165 

0.000  001 

0.000  000  165 

0.000  02 

A?  » 0.999  995 

0.120 

0.000  005 

0.000  000  600 

0.000  08 

A.  » 0.999  99 
o 

0.065 

0.000  01 

0.000  000  650 

0.000  09 

Ag  » 0.999  9 

0.0425 

0.000  1 

0.000  004  25 

0.000  57 

*10“  °-999 

0.0275 

0.001 

0.000  027  5 

0.003  67 

A11»  0.99 

0.0125 

0.01 

0.000  125 

0.016  70 

\2  = 0.9 

0.01025 

0.1 

0.001  025 

0.136  97 

*P(B/Ai)  for  test  failure  is  given  by  1-Ai  where  Ai  is  the  cell  value*  Therefore/ 
P(3/Ax)  - 1-A±  =■  1-0.999  999  995  - 0.000  000  005  = 5xl0~9. 

If  the  test  had  been  a success  P(B/Ai)  is  given  by  Ai . 
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TABLE  (3), 


Bayesian  updating  calculations  if  first  test 
results  in  a failure  (contd) • 


Cell  Values, 

P(Ai) 

p (B/Aiy  * 

■ 

P (B/Ai)  P (Ai) 

p (ai/b)  — xya).. 

A13  **  0,8 

0.0055 

0.2 

0,001  1 

0.146  99 

A14  * 0,7 

0.001 

0.3 

0.000  3 

0.040  09 

*15  ■ °*6 

0.001 

0.4 

0.000  4 

0.053  45 

*16  “ °-5 

0.001 

0.5 

0.000  5 

0..066  82 

A = 0,4 
17 

0.001 

0.6 

0,000  6 

0.080  18 

A =0.3 

0.001 

0.7 

0.000  7 

0.093  54 

18 

A19  " °*2 

0.001 

0.8 

0.000  8 

0.106  90 

a„a  * 0.1 
20 

0.001 

0,9 

0.000  9 

0.120  27 

A = 0.0 

0.001 

1.0 

0.001  0 

0.133  63 

21 

P (B)  = 

l P(B/Ai)P(Ai) 

0.0074833 

1=1 

*P(B/Ai)  for  test  failure  is  given  by  1-Ai  where  Ai  is  the  cell  value.  Therefore, 
P { B/Ai } = 1-Ajl  * 1-0.999  999  995  « 0.000  000  005  « 5xl0“9. 

IF  THE  TEST  HAD  BEEN  A SUCCESS  P (B^Af  ) IS  GIVEN  BY  A;, 


P(A./B)„  (0,00325)  (0.000  000  006) 

1 

P(A*/B)_  1.625  X 10-11  _ 1.216  X 10  “-1 3 

1 0.0Q7W"  _ 

For  cell  value  Am 

From  Table  1,  Ago  * 0.999 

From  Table  2,  P(A10)  - 0.027.5 

We  want  to  compute  P(A^q/B), 

The  Bayesian  Calculations  are: 

P(Alr./B)  P(A10)  P (H/A10) 



n=21 

1J(B)  p(B/Ai)  P(A,)  - 0.007)i833 

P(Ain/B)  - (0.0275)  (0.001)  0.000275 

(0.00716333) 0T007£SJ3 

P(A1Q/B)  - 0.00367 

Table  3 shows  the  -results  of  the  remaining  P(Ap/B)  Bayesian  calculations. 
These  P.(A_. )?  ?(),  and  P(A^/B)  = P,  are  plotted  in  figure  2.  The  posterior 
distribution  is  used  as  the  new  prior  for  the  next  test  (see  Tab!'-'  It). 

Figure  2 shows  a histogram  of  the  prior  distribution  updated  by  the 
Bayesian  calculations  after  a single  test  resulted  in  a failure.  A 
shift  in  the  P(A.)  probabilities  is  observed  with  the  higher  reliability 
values , 0.999  99$  9 and  0.999  999  5,  being  shifted  to  0.9  and  0.-3,  -respectively 
Also  significant  values  of  reliability  occur  between  0*7  down  to  0.0, 
and  the  updated  distribution  shows  a trend  clustered  near  zero.  This  section 
indicates  that  ©▼•none  initial  test  failure  way  radically  alter  a prior 
distribution . ^ 

The  results  of  Bayesian  updating  calculations  for  a second  test 
resulting  in  a success  after  an., initial  failure  is  shown  in  example 
2,  histogram . figure  3 and  table  5* 


'"It  should  be  mentioned  that  the  case  of  more  than  one  failure  in  a 
two  hour  tost,  is  still  to  be  treated  as  one  failure  in  tins  approach.  In 
oilier  words,  either  the  system  functions  properly  or  fails  at  the  end  of 
the  two  hour  test. 
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1 


G666G  *0 


(V)  d - Aimavaoad 


P ( A-i  ) : Prior  : P (A~- ) 


r Pi i ip.'  ) 


0,00323 

0.000 

:)<. 

1 0,01300 

0 . uuD 

00 

L. . O30OU 

0.000 

00 

to  2 3.300 

0.000 

/ \ 1 
'.J'j 

O.Pltpbo 

0,000 

02 

t) . 16iiu0 

0 . oOu 

02 

0,12000 

0,000 

-ad 

0 , 06300 

f .000 

09 

0.01*230 

. O00 

1 \ 

0.0273'u 

0 . U03 

67 

u. 01230 

0.OI6 

70 

0 ,0V  >2:-> 

•.j . 1 3'0 

97 

u « 

0 . Ido 

JV 

0.  ,oo.loo 

0 . 0d<  ■ 

09 

0.00100 

0 . o?  3 

d3 

0.00100 

{ , 006 

02 

0,001.00 

0 , ',a)0 

Id 

0.00100 

0.093 

0,00100 

u . 106 

90 

O.OOlUG 

0.120 

27 

0,00100 

i — 
ao 
eu 

63 

HOT  Li:  The  accuracy  ox  these  Pay  'Sian  calculations  n ■”> 


carried  out.  to  thirtr 


oimal  ,)lao 


i rj  coYrtpute.! 


i.a  need  In.  subsequent  oa.lcv’latjons  ‘>n  thirteen  Herd  »ials 
although  ti'ie  .results  are  only  11  stod  as  live  decimals  1 
tables . 
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JA5E2  5*  Bayesian  updating  calculations  if  second  test  results  in  a 
success  after  an  initial  failure. 


Cell  Values , 

P(Ai)  is 
P(Ai/B)  of 
Table  (3) 

0 

P {B/Ai) * 

1 

P (B/Ai)  P(Ai) 

P (Ai/B)  - P (Aj)P (B/Aj) 

0.999  999  995 

0.000  00 

0.999  999  995 

0.000  00 

0.000  00 

0.999  999  99 

0.000  00 

0.999  999  99 

0.000  00 

0.000  00 

0.999  999  95 

0.000  00 

0.999  999  95 

0.000  00 

0.000  00 

0.999  999  9 

0.000  00 

0.999  999  9 

0.000  00 

0.000  00 

0.999  999  5 

0.000  02 

0.999  999  5 

0.000  02 

0.000  04 

0.999  999 

0.000  02 

0.999  999 

0.000  02 

0.000  04 

0.999  995 

0.000  08 

0.999  995 

0.000  08 

0.000  18 

0.999  99 

0.000  09 

0,999  99 

0.000  09 

0.000  20 

0.999  9 

0.000  57 

0.999  9 

0.000  57 

0.001  27  ' 

0.999 

0.003  67 

0.999 

0.003  67 

0.008  18 

0.99 

0.016  70 

0.99 

0.016  53 

0.036  84 

*P(B/A^)  for  a test  success  if  given  by  the  A^  cell  values. 
If  this  second  test  had  been  a failure  then 
P (B/A^)  is  given  by  (1-Ai) . 


213 


TABii^  5,  Bayesian  updating  calculations  if  second  test  results  in  a 
success  after  an  initial  failure  (contd) . 


Cell  Values , 

P(A±)  is 
P<Ai/B)  of 
Table  (3) 

PtB/Ai)* 

PtB/Ai)  p(AjL) 

P (**}  (Ai)P  (B/Ai: 

P(B) 

0.9 

0.136  97 

0.9 

0.123  27 

0.274  59 

CO 

• 

o 

0.146  99 

o 

. 

00 

0.117  60 

. 0.261  94 

0.7 

0.040  09 

0.7 

0.028  06 

0.062  51 

0.6 

0.053  45 

0.6 

0.032  07 

t 0.071  44 

0.5 

0.066  82 

0.5 

0.033  41 

0.074  42 

0.4 

0.080  18 

0.4 

* 0.032  07 

0.071  44 

0.3 

0.093  54 

0.3 

" 0.028  06 

0.062  51 

0.2 

0.106  90 

o 

. 

to 

0.021  38 

0.047  63 

0.1 

0.120  27 

0.1 

0.012  03 

0.026  79 

0 

0.133  63 

0.0 

0.000  00 

, 

0.000  00 

P(B) 

i 1 p{B/Ai)p(Ai) 

* 0.448  93 

*P(B/Aj)  for  a test  success  if  given  by  the  cell  values. 
If  this  second  test  had  been  a failure  then 

P(B/A^)  is  given  by  (1-AjJ  . 


EXAMPLE  2.  The  second  test  results  in  a success  after  an  initial 
failure.  The  Bayesian  sample  calculations  follow  for  A|  and  A-| , , cell 
values  using  Pp  (A^)  as  the  new  prior  probability, 

];"or  dell  Value  Aj.  From  Table  a , I\  (Ag)  = 0,000  00,  b i nee  we  cannot 
use  zero,  we  must  use  the  number  calculated  from  example  which  is 
P ( A ] /B ) = 1,216  x 10  3 * Pg(Ag),  The  new  posterior  distribution  is  now 

calculated: 

l’Ul/B)  _ P(A]_)  P (P/Ag 

pTb) 

P(B/A1)  « 0,999  999  993' 

From  Table  $ , P(B)  = O.I1I4893 

r'(JlL/li)_  P:1(A1)  P (3/A-,)-  y.  rn  -13)(o  ,999  900  pop) 

~ ~~ — fou — “ (ojmm 

P(Ag/B)  — 9.709  X ICT13  (shown  as  0.000  00  in  Table  3) 

For  Cell  Value  Ain*  From  Table  )j , P -\  (Alo)  = 0.00367*  The  new  posterior 
(or  final)  value  for  P (Agn/B)  is  now  calculated  for  cell  Agp,  - 0.999  and 
a second  test  resulting;  in  a success. 

P(Aio/B)^  PptApo)  P (B/A10) 
p{b) 

From  Table  3,  P(B)  = 0 . J4J 4 8 9 3 ami  P(B/A1q)  is  0.999  because  second  test 
was  a success. 

P(Ai n/B)  = (0.00367)  (0.999)  - 0-008.1.8 
(0. 1(393) 

Table  !.>  shows  the  results  of  the  remaining  P(Ag/B)  Bayesian  calculations 
w.iicii  are  plotted  in  histogram  form  in  figure  3 ■ along  with  its  prior.  By 
observing  this  histopram,  one  can  see  that  the  reliability  cells  are  now 
s, li  fted  away  from  0.0  and  a sign.fl can t increase  in  relative  probability 
values  is  observed  at  0.9  and  0,8, 

Table  6 shows  the  flow  of  data  for  each  cell  from  the  Prior  Distribution 
to  second  test  a success  following  a first  test  failure. 
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TABLE  6.  Prior  and  Final  Distributions  Alter  Second  Tret  is  a 
Success  and  First  Test  was  a Failure. 


PRIOR  DISTRIBUTION 
P0(A, ),  x“l  to  21 
Figure  1 

FIRST  fi’FS?  A FAILURE 

fAc  o 

SECOND  TEST  A SUCCESS 
(A,) 

Figure  3 

0.00 3 On 

0.000  00 

0.000  O'. 

0.01300 

0.000  00 

0.000  Ou 

0.0^000 

0.000  00 

0.000  00 

0.23300 

0.000  00 

0.000  00 

0. 2h?S) 

0.0 Ol.J  02 

0 , '.A  X)  0)4 

0 . 16.500 

0.000  02 

0.000  oh 

0.12000 

.000  od 

0.000  id 

0.06300 

0.000  09 

0 .000  20 

o.olpio 

u.000  5? 

0.001  27 

0.02730 

0.003  6? 

0.006  Id 

0 .01230 

o.oio  70 

0.036  S4 

0 .01023 

0.136  9 7 

o.?7ii  ;:> 

0.00.330 

(.) . 1J46  99 

0.261  ;!i 

0.C0100 

c.olio  09 
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7.0  VARIOUS  PRIOR  DISTRIBUTIONS  AS  APPLIED  TO  DIFFERENT  SETS  OF  TESY 
DATA.  Other  initial  prior  distributions,  termed  "AM , are  chosen  in  order 
to  indicate  how  sensitive  the  selection  of  prior  failure  distributions 
are  to  the  final  resultant  distributions  after  Bayesian  updating  calculations . 
In  other  words,  the  following  histograms  are  used  to  illustrate  the  degree 
change  of  prior  distribution  that  occurs  after  Bayesian  updating  as  new 
reliability  test  data  becomes  available.  Table  7 gives  the  reliability 
values  at  each  reliability  cell,  A±,  for  the  following  prior  distributions: 
Binomial  type  average  distribution  of  table  2 ; Uniform  distribution; 

Peaked  (at  l*th  cell  mid  value)  distribution;  Peaked  (at  7th  cell  mid 
value)  distribution;  and  Skewed  Binomial  type  distribution.  Figures  1, 

U,  5,  6,  and  7 show  relative  frequency  histogram  graphs  of  each  of  these 
prior  distributions. 

The  tables  and  histoggaa  graphs  of  Appendix  A will  be  used  to 
illustrate  the  change  that  occurs  in  the  Bayesian  estimates  as  new 
reliability  test  data  becomes  available.  The  Bayesian  calculations  were 
performed  with  an  EAI  81|00  digital  computer  with  a double  precision 
calculation  accuracy  to  thirteen  decimal  places.*  This  accuracy  is 
necessary  because  it  is  important  to  use  extremely  small  numbers  for 
each  Bayesian  calculation  without  rounding  them  to  zero. 

The  results  of  Bayesian  updating  calculation  of  ten  tests  with 
various  success/failure  combinations  for  various  initial  prior  distributions 
are  shewn  in  tables  A,1  to  A. 5 and  in  histogram  figures  A.lto  A. 25  inclusive. 
In  the  cases  where  no  failures  occurred  in  ten  tests  (figures  A.l,  A. 6, 

A, 11,  A. 16,  A. 21  it  can  be  observed  that  the  final  histogram  data  (solid 
lines)  is  generally  shifted  to  the  left  or  higher  reliability  regions  with 
a slight  increase  in  the  relative  probability  values  at  the  higher 
reliability  cell  values.  It  can  be  seen  that  the  selection  of  prior 
distributions  (dotted  lines)  biases  the  results  of  the  Bayesian  calculations 
if  all  feen  tests  are  successes,  and  the  final  histogram  looks  very  similar 
to  the  prior. 

The  Bayesian  calculations  which  updated  the  Binomial  Type  Prior 
Distribution  for  ten  two -hour  tests  with  various  success/failure  combinations 
are  in  table  A.l  and  are  plotted  in  figures  A.l  to  A. 5 inclusive.  Figure 
(A. 2)  shows  the  updated  Bayesian  results  (solid  lines)  of  one  failure  in 
ten  tests  upon  this  binomial  prior  (dotted  lines).  One  would  intuitively 


*See  Appendix  B for  Computer  Program 
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expect  the  reliability  distribution  to  be  centered  around  0.9  whioh  is 
clearly  Indicated  In  the  figure.  Similarly,  for  the  Bayesian  result* 
of  two  failures  in  ten  tests  one  would  expect  the  reliability  point 
estimate  to  be  near  0*8.  For  two  failureo  in  ten  testS|  figure  (A»3) 
shows  most  of  the  data  to  be  distributed  around  0.9  beouse  the  prior 
distribution  biases  the  final  distribution  towel's  itself.  Since  the 
prior  distribution  is  . the  high  reliability  region , in  this  case  the 
final  distribution  did  not  completely  eliminate  the  prior  with  only 
ten  teata.  For  fire  failure!  in  ten  tests,  the  inference  ie  that  moat 
of  the  final  distribution  should  be  near  the  reliability  point  estimate 
of  0.5.  Figure  (A«U)  showa  the  final  distribution  for  five  failures 
in  ten  tests,  Most  of  the  data  la  distributed  about  the  0.5  cell 
reliability  value  with  a secondary  relative  probability  peak  at  0.8  due 
to  the  biasing  or  wsighting  oaussd  by  the  prior  distribution.  Ths 
histogram  in  figurt  (A.5)  shows  the  results  of  tsn  failures  in  tsn  tests. 
Hers  ths  inference  is  obvious  since  most  of  ths  final  distribution 
should  bs  about  the  0,0  oell  reliability  value,  Thus,  tm  failures  in 
ten  tests  have  radically  altered  the  prior  distribution,  as  ebofcld  be 
foqpeoted. 

Table  A,1  ahows  the  result*  of  Baysaian  calculations  upon  ths 
following  Iseta  IF  ♦ 93  (one  failure  followed  by  nine  successes), 

!i3  + l?  + 5fl  (four  aucosassa  followed  by  one  failure  and  than  five 
•uooeaaea),  and  98  ♦ IF  (nine  successes  followed  by  one  failure).  In 
gamerel,  the  results  of  the  Bayesian  calculations,  given  ang,  on*  prior, 
are  the  same  with  any  equal  number  of  total  teste  regardless  of  the 
order  in  which  the  teat  data  la  received. 

The  Bbyeaian  result*  whioh  update  the  Uniform  Prior  Distribution 
for  ten  two -hour  teata  with  various  auoooss/fallure  combinations  are 
|ivsn  In  tabla  A *2  end  ars  plotted  in  histogram  form  in  figures  A.6  to 
A, 10  inolusivt.  The  finsl  histogram  data  for  this  prior  is  distributed 
around  the  reliability  point  estimates  which  ons  would  sxpsot  to  obtain 
for  tha  particular  tsat  results  indicated. 

The  Bayesian  results  for  the  Peaked  (st  fourth  osll)  Prior 
Distribution  sr*  given  in  table  A, 3 and  plotted  in  figursa  A, 11  to 
A, 15  inoluelve  for  ten  testa  with  various  suocsss/failurs  combinations. 
Again  tha  final  histogram  data  for  this  prior  is  distributed  around  ths 
reliability  point  estimates  or  class  intervals  whioh  one  would  expeot 
for  the  test  results  given. 
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Table  A.l|  and  figures  A.l#  to  A. 20  inclusive  show  the  Bayesian 
results  for  the  Peaked  (at  seventh  cell)  Prior  Distribution  for  ten 
tests  with  various  success/f allure  combinations.  The  final  histogram 
data  is  again  distributed  around  the  cell  reliability  values  expected 
except  for  figure  A. 17.  This  final  distribution  for  one  failure  in  ten 
tests  shows  approximately  2.5  percent  of  the  data  distributed  in  cell 
value  0.999  995  due  to  prior  distribution  biasing. 

For  the  Skewed  Binomial  Prior  Distribution  with  various 
success/failure  combinations  of  ten  tests,  the  Bayesian  results  are 
given  in  table  A. 5 and  are  plotted  in  figures  A. 21  to  A. 25  Inclusive. 

The  final  histogram  data  is  distributed  as  would  be  expected  except  for 
the  case  of  five  failures  in  figure  A.2h.  Here  the  data  is  slightly 
biased  by  the  prior  to  the  left  towards  the  prior. 

To  illustrate  the  influence  of  the  prior  distribution  upon  the 
final  results  of  the  updating  Bayesian  calculations  and  estimates  from 
new  reliability  test  data,  consider  next  the  results  of  one  failure  in 
100,  500,  1000,  5000,  and  10,000  tests  for  each  prior  distribution. 

The  Bayesian  results  of  these  larger  number  of  tests  are  given  in  tables 
A. 6 to  A. 10  and  histogram  figures  A. 26  to  A.50  inclusive.  It  can  be 
observed  that  the  prior  distribution  in  most  cased  in  this  paper  does  not 
bias  the  final  histograms  to  any  large  degree  except  in  figures  A. 32, 

A, 33*  A.3U*  A. 35,  and  A. 39  to  A.U5  inclusive.  L<£us  briefly  summarize 
the  resfclts  of  this  test  data  upon  our  various  prior  distributions. 

Given  a binomial  prior  distribution  (figure  1)  with  the  most 
probable  cell  at  0.999  999  5*  it  can  be  observed  that,  as  the  number 
of  tests  increase  and  the  percent  defective  decreases  (figures  A. 26, 

A.27f  A. 28,  A. 29*  A.30)  the  final  or  posterior  distribution  will  be 
gradually  shifted  towards  the  ordinate  or  in  the  direction  of  increasing 
reliability.  Concurrently,  the  most  probable  cell  value  of  the 
posterior  distribution  will  change  from  0,99  to  0.9999.  However,  if 
our  test  results  indicate  poor  reliability  (far  to  the  right  of  the 
binomial  prior)  the  binomial  prior  will  be  almost  entirely  washed  out. 

Given  a uniform  prior  distribution  (figure  U)  it  can  be  observed 
that  as  the  number  of  tests  increase  and  the  percent  defective  decreases 
(figures  A. 31*  A. 32,  A. 33,  A,3U,A.35)  the  posterior  distribution  will 
be  gradually  shifted  towards  the  ordinate  or  in  the  direction  of  increasing 
reliability.  Concurrently,  the  most  probable  cell  of  the  posterior  distribution 
will  change  from  .99  to  .999  995.  However,  if  the  test  results  indicate 
poor  reliability  (far  to  the  right  of  the  uniform  prior) , the  uniform 
prior  will  be  reduced  in  amplitude  or  almost  refuted.  There  are  instances 
where  two  peaks  appear  in  the  posterior  or  final  distribution.  This  is 
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caused  by  the  biasing  of  the  uniform  prior,  particularly  in  the  range 
of  high  reliability.  In  these  cases,  the  test  data  results  are 
starting  to  reinforce  the  last  uniform  prior  cells. 

Given  a peaked  (at  lith  cell)  prior  distribution  (figure  5) 
it  can  be  observed  that  as  the  number  of  tests  increase  and  the 
percent  defective  decreases  (figures  A.36lf  A. 37,  A. 38*  A, 39*  A.1±0) 
the  posterior  distribution  will  be  gradually  shifted  towards  the 
ordinate  or  in  the  direction  of  increasing  reliability.  Assuming  that  the 
reliability  increases  during  testing  and  approaches  the  fourth  cell 
(0.999  999  9)* the  peaked  prior  distribution  is  reinforced.  Concurrently, 
the  most  probable  cell  of  the  posterior  distribution  (cell  with  highest 
reliability)  will  change  from  0.99  to  0.999  999  9.  For  the  5000  and  10,000 
test  cases,  the  large  jump  of  one  posterior  or  final  peak  to  0.999  999  9 
(or  the  fourth  cell)  is  the  result  of  the  selection  of  the  prior  with 
most  of  the  relative  probability  or  amplitude  in  the  fourth  cell.  Here 
the  test  data  inferences  or  results  are  starting  to  reinforce  the 
prior  distribution.  Finally,  if  the  test  results  indicate  poor  reliability 
(that  is,  far  to  the  right  of  the  fourth  cell),  the  peaked  prior  will  be 
totally  washed  out. 

Given  a peaked  (at  7th  cell)  prior  distribution  (figure  6)  it 
can  be  observed  as  the  number  of  tests  increase  and  the  percent 
defective  decreases  (figures  A. hi,  A.)|2,  A.U3,  A.UU,  A.i*5)  the  posterior 
distribution  will  be  gradually  shifted  towards  the  ordinate  or  in  the 
direction  of  increasing  reliability*  Assuming  the  reliability  increases 
during  testing  and  approaches  the  seventh  cell  (0.999  995)  the  peaked 
prior  distribution  is  reinforced.  Concurrently,  the  most  probable  cell 
of  the  posterior  distribution  will  change  from  0.99  to  0.999  995*  The 
centering  of  posterior  information  or  data  in  the  seventh  cell  (.999  995) 
is  the  result  of  the  prior  being  placed  almost  entirely  in  the  seventh 
cell,  and  the  fact  that  the  test  data  inferences  are  starting  to  reinforce 
the  prior.  Also,  if  the  test  data  results  indicate  poor  reliability 
(that  is,  far  to  the  right  of  the  seventh  cell),  this  peaked  prior  will 
be  totally  washed  out. 

Given  a skewed  binomial  prior  distribution  (figure  7)  with  the 
most  probable  cell  at  0.999  999  95,  it  can  be  observed  that  as  the  number 
of  tests  increaseiand  the  percent  defective  decreases  (figures  A.U6, 

A.U7,  A.U8,  A .h9,  A.50)  the  posterior  distribution  will  be  gradually 
Shifted  towards  the  ordinate  or  in  direction  of  increasing  reliability. 
Boncurrently,  the  most  probable  cell  of  the  posterior  distribution  will 
change  from  0,99  to  0.999  9.  However,  if  our  test  results  indicate  poor 
reliability  (that  is,  far  to  the  right  of  the  prior  cells),  then  the 
skewed  binomial  prior  will  be  almost  entirely  washed  out. 
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In  order  to  indicate  the  Bayesian  results  of  no  failures  for 
various  test  data  upon  several  prior  distributions,  tables  A. 11, 

A. 12,  and  A *13,  and  figures  A.  51,  A* $2,  and  A. £3  have  been  generated. 

The  conclusion  is  that  the  final  histograms  look  very  similar  to  the 
prior  distribution  if  all  tests  are  successes  and  the  number  of  tests 
are  not  greater  than  the  indicated  cell  reliability  bands  order  of  magnitude. 
In  other  words,  if  the  reliability  of  the  sample  tested  is  greater  than 
the  most  probable  cell  or  class  interval  of  the  prior  distribution, 
then  the  final  distribution  will  be  distributed  around  the  reliability 
of  the  sample  tested.  For  example,  if  the  most  probable  cell  of  the 
prior  is  $).99  and  10,000  tests  are  completed  with  no  failures),  one  can 
expect  the  final  distribution's  most  probative  cell  value  to  be  0*999  9. 

The  following  conclusions  can  be  reached  concerning  the  Bayesian 
Discrete  Reliability  inference  procedure*  In  the  discrete  reliability 
cell  formulation,  the  choice  of  the  number  of  cells,  the  location  of 
the  cell  reliabilijty  values,  and  cell  boundaries  are  all  arbitrary  when 
coverfcAggthe  complete  reliability  scale  of  numbers  from  zero  to  unity 
and  can  be  chosen  to  suit  the  user.  The  time  for  each  test  is  picked 
to  be  equal  to  the  defined  or  specified  equipment  reliability  time 
interval.  For  example,  if  we  choose  or  require  our  equipment  or 
mission  reliability  to  be  0.99  per  hour,  then  each  test  or  data  point 
is  to  be  one  hour.  If  we  require  a reliability  of  0.98  for  three  hours 
of  operation,  then  each  test  is  to  be  three  hours  long.  If  more  than  one 
failure  is  recorded  in  our  test  time  interval,  the  failures  are  still  to 
be  treated  as  one  failures  In  this  approach  either  the  system  functions 
properly  or  fails  during  our  test  time  interval.  The  final  or  posterior 
distribution  will  not  be  affected  by  the  order  of  failure  occurrence. 

In  other  words,  the  Bayesian  results,  given  any  one  prior,  are  the 
same  with  the  same  total  number  of  tests  regardless  of  the  order  in  which 
the  test  data  is  received. 

An  important  issue  that  must  be  resolved  in  this  application  of 
the  Bayes'  Technique  is  the  choice  of  an  appropriate  prior  distribution. 

From  this  report,  it  is  clear  that  the  prior  distribution  can  bias  the 
final  distribution  in  certain  cases.  From  the  histograms  it  is  evident 
that  the  least  biasing  type  of  prior  distribution  is  the  binomial  type. 

There  are  some  questions  that  must  be  answered  in  order  to  postuAAte 
appropriate  and/or  usable  prior  distributions.  What  does  the  prior 
distribution  look  like?  IS  it  to  be  a continuous  or  discrete  one?  For  a 
particular  choice  of  prior  distribution,  can  a bound  be  placed  on  the  error 
(in  the  probability  sense)  one  can  expect  in  the  final  distribution?  In 
the  implementation  of  the  Bayesian  statistics  in  reliability  testing,  can 
agreement  be  reached  between  the  contractor  and  government  on  the  selection 
of  an  approriate  prior  distribution!  These  are  just  a few  of  the  questions 
that  should  be  answered  before  a prior  distribution  is  to  be  selected. 

In  this  Bayesian  discrete  approach  the  final  distribution  is 
available  for  examination  after  each  test.  In  other  words,  the  final 
distribution  can  be  observed  after  each  laboratory  and  experimental  tests 
(each  prototype  and  production  flight  tests)  and  updated  at  each  of  the 
program  phases  on  a continuing  basis.  Finally,  flight  test  operational 

data  can  be  used  by  the  Bayesian  method  to  update  the  pr*v*ioa  final 
(posterior)  reliability  distribution. 
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The  final  distribution  Pp  i s new  used  as  the  prior  di  st.ri  bard  or  ea  l 1 ed 
Pj  ,;i  to  bo  updated  bv  prototype1  .laboratory  test  failure  data  rose  Filip; 
irtr  now  final  distribution  Pp.  Phis  procedure-  is  eon  tinned  on  r.n 
iterative  basis  usl.ne;  acceptance  and  production  test  data  to  r;  dale 
each  prior  distribut'd on.  In  other  words,  the  final  or  posterior 
r< 'liabili  ty  distribution  can  bo  obtained  after  each  laboratory  and 
experimental  test,  each  prototype  arid  production  fl’pht  test,  and  updated 
at  oa'c  h of  the  propram  phases  on  a conti  nrinp  basts. 
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POSTERIOR  DISTRIBUTIONS  DERIVED  FROM  VARIOUS 
HYPOTHETICAL  PRIOR  DISTRIBUTIONS 
AND  TEST  DATA 
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Cell  Reliability 
Values  A- 


IDS 


10  Two  Hour  Tests  with  Binomial  Type  Average  Prior  Distribution  of  Table  Z 


IF 

+9S 


4S 

9S 

2F 

3S 

8S 

5F 

+1F+5S 

+ 1F 

+ 8S 

+2F+5S 

+2F 

+5S 

Tests 

Tests 

Tests 

Tests 

Tests 

Tests 

Tests  Tests 


0.0033 


0.0133 


0.0511 


0.2401 


0.1686 


0.1226 


0.0664 


0.0434 


0.0006 


0.0000 


0.0000 


0.0000 


0.0000 


0.0000 


002 


0.0002 


0.0008 


0.0009 


0.0060 


0.1610 


0.0057 


0.0014 


0.0002 


0.0000 


0.0000 


0.0000 


0.0000 


0.0000 


0.0000 


0.0000 


0.0000 


0.0000 


.0000 


0.0000 


0.0000 


0.0003 


.0107 


0.0026 


0.0000 


0.0000 


0.0000 


0.0000 


0.0000 


0.0000 


0.0000 


0.0000 


0.0000 


0.0146 


0.1395 


0.0988 


0.0988 


0.0014 


0.0000 


0.0000 


0.0000 


0.0000 


0.0000 


0.0000 


0.0000 


0.0000 


0.0000 


0.0000 


0.0001 


0.0007 


0.0189 


0.0720 


0.6705 


S = Success 
F * Failure 


Table  A.l  Bayesian  Calculation  Results  of  Ten  Two  Hour  Tests  with  Various 
Success/Failure  Combinations,  for  Binomial  Type  Prior. 
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CELL  RELIABILITY  VALUES  finrl 

> FIGURE (fl. 5). BINOMIAL  PRIOR. A. RESULTS  OF  TEN  TESTS. TEN  FAILURES  A+lOF 
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ro 

u> 

■p- 


f 10  Two 

Cell  Reliability 
Values  Ai 

10S 

Tests  ; 
" - .— | 

IF 

+9S 

Tests 

0.999  999  995 

0.1419 

0.999  999  99  | 

0.1419 

0.999  999-95 

0.1419 

0.999  999  9 

0.1419 

0.999  999  5 

0.1419 

0.999  999 

0.1419 

0.999  995 

0.1419 

0.999  99 

0.0018 

0.999  9 

0.0014 

; . 0.999 

0.0014 

0.99 

HRHI 

; 0.9 

0.0005 

■ 

0.8 

0.0002 

0.7 

0.0000 

! 0.6 

0.0000 

■ 0.5 

0.0000 

- 0.4 

0.0  000. 

0.3 

0.0000 

0.2 

0.0000 

0.1 

0.0000 

0.0 

0.0000 

1 

S 5 Success  Table  A*2 


F 


Failure 


Hour  Tests  with  Uniform  Prior  Distribution 


4S  9S 

+1F+5S  +1F  l +8S  | +2F+5S [+2F  | +5S 

Tests  Tests  Tests  Tests  Tests  Tests 


0.0000 


0.0000 


0.0001 


0.0001 


0.0011 


0.0053 


0.0001 


0.0011 


0.0106 


0.4131 


0.2863 


0.1291 


0.0430 


0.0104 


0.0017 


0.0001 


0.0Q00 


+5F  10F 

Tests  Tests 


0.0000 


0.0000 


0.0000 


0.0000 


0.0000 


0.0000 


0.0000 


0.0000 


0.2128 


0.3317 


0.2565 


0.1328 


0.0483 


0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0016 


0.0291 

0.0000 

0.1132 

0.0000 

0.2207 

0.0001 

0.2707 


0.0117 

0.2207 

0.0016 

0.1132 

0,0001 

0.0291 

0.0000 

0.0016 

0.0000 

0.0000 

0.0007 


0.0041 


0.0189 


0.0720 


Bayesian  Calculation  Results  of  Ten  Tests  with  Various 
Success/Failure  Combinations,  for  Uniform  Prior. 
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CELL  REL1RBILITT  VALUES  mnhl 

FIGURE (R. 91. UNIFORM  PRIOR. fl. RESULTS  OF  TEN  TESTS. 5 FAILURES  FUSE -5S 
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Cell  ReHablUt 
Values, 

0.999  999  995 
0.999  999  99 

0.999  999  95 


0.999  999 
0.999  995 


| 0.2 


0.0 

S 2 Success 


10S 

Tests 

IF  4S 

+9S  +1F+5S 

Tests  Tests 

0.0020 

0.0000 

0.0020 

0.0000 

0.0020 

0.0000 

0.9788 

0.0005 

0.0020 

0.0000 

0.0020 

0.0000 

0.0020 

0.0001 

0.0020 

0.0001 

0.0020 

0.0011 

0.0020 

0.0106 

n nm  a 

0.0981 

Q . 0007 

0.4159 

0 , 0000 

0.0000 

0.0000 


9S  2F 

+1F  +8S 

Tests  Tests 


ith  Cell 1 Prior  Distribution 

3s  as  5?  rss 

+2F+5S  +2F  +5S  +5: 

Tests  Tests  Tests  Te 


Tests  Tests 


0.0433 
0.0105 
0.0017 
0.0001 
0.0000 
0.0000 
0 . 0000 


0.0000 

0.0000 

0.0000 

0.0000 

JLJHLOIL 

» 

0.0  OOP' 

0.0000 


0.0000 

0.0046 

0.2128 


0.2565 

. Q.1.L32  L 

0.0483 

Q.OU  7 

0.0016 

0.0001 

0.0000 

0.0000 


0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0016 

0.0291 

0.1132 

0.2207 

0.2707 

0.2207 

0.1132 

0.0291 

0.0016 

0.0000 


jO.OOOO 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0 . 0000 

0.0000 


0.0007 

0.0041 

Q.Ql.aSL 

0.0720 

0.2338 

0.6705 


Table  A«3  Bayesian  Calculation  Results  of  Ten  Tests  with  Various 

Success/Failure  Combinations,  for  Peaked  (at  4th  Cell)  Prior. 


F = Failure 
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FIGURE  IH.  13) . PEAKED [4TH  CELL.) PRIOR.  H. RESULTS  OF  10  TESTS, 2 FAILURES. H+2F+85 
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Cell  Reliability 
Values  Ajl 

0,999  999  995 
0.999  999  99 


10  Two  Hour  Tests  With  Peaked  (at  7th  Cell)  Prior  Distribution 


IF  4S  9S  2F  3S  8S  5F  5S 

10S  +9S  +1F+5S  +1F  +8S  +2F+5S  +2F  +5S  +5F  10F 

Tests  Tests  Tests  Tests  Tests  Tests  Tests  (Tests  Tests  Tests 


S s Success 


Table  A*U  Bayesian  Calculation  Results  of  Ten  Tests  with  Various 
' Success /Failure  Combinations,  for  Peaked  (at  7th  Cell)  Prior 


F 5 Failure 
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2b3 


nw 


X 


10  Two  H 

our  Tes 

ts  With  1 

pproxima. 

te  Skewed  Binomial  Prior  Distribution 

Ceil  Reliability 
* Values  A.. 

10S 

Tests 

IF 

+9S 

Tests 

4S 

+1F+5S 

Tests 

9S 

+1F 

Tests 

2F 

+8S 

Tests 

3S 

+2F+5S 

Tests 

8S 

+2F 

Tests 

5F 

+5S 

Tests 

5S 

+5F 

Tests 

10F 

Tests 

0.999  999  995 

0.0010 

___  . 0.0000 

0.0000 

0.0000 

0.0000 

0.999  999  99 

0.0010 

n norm 

0.0000 

0.0000 

0.0000 

0.999  999  95 

0,4977 

rasiiiiaa 

n nnnn 

0.0000 

n nnnn 

i i 

0.0000 

MPIPIHDi 

0.0001 

• 0.0000 

0.0000 

0.999  999 

0,0684 

0.0001 

0.0000 

0.0000 

0.0000 

0.999  995 

0.0487 

0.0003 

0.0000 

0.0000 

0.0000 

Cua.99,99 

UliBBBi 

0.0004  

0.0000 

0.0000 

| 

0.999  9 

0.0024 

0.0000 

0.0000 

~ 0- 399 

HMMi 

0.01 R0 

. 0 99 

0.0103 

0.1215 

: 0.9 

0.0036 

0.46R5 

0.2976 

0.0050 

j 0.8  

H^IISBI 

0.2921 

0 4176 

1 

0.7 

0.0732 

0.1  794, 

0.1733 

0.6 

0.0195 

0.0743, 

0.2703 

0.0003 

-0.5 

0.0035 

0.0203 

0.2486 

0.0018 

0.4 

RBPSBI 

n . nnna 

0.0033 

0.1 361 

0.3 

0.0000 

0.0693 

IRREHI 

1 

0.2 

0.0000 

0.1313 

0.0000 

0.0000 

0.0000 

0.2132 

0.0 

0.0000 

0.0000 

0.0000 

0.0000 

0.6115 

Bayesian  Calculation  Results  of  Ten  Tests  with  Various  - 
Success/Failure  Combinations , for  Skewed  Binomial  Prioj: 


S s Success 
F 3 Failure 
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ro 

-f=- 

Co 


J 

Testa  Witt 

i Binomial  Type  Average  Prior  Distribution  of  Table  2 

Cell  Reliability 
Values 

99S  + IF 
or  IF  + 99S 

499S  + IF 
or  IF  + 49 9S 

1 

999S  + IF 
or  IF  + 999S 

4999S  + IF 
or  IF  + 4999S 

9999S  + IF 
or  IF  + 9999S 

0.999  999  995 

0.0000 

0.0000 

0.0000 

0,0000 

0 .0000 

0.999  999  99 

0.0000 

0.0000 

0.0000 

0.0000 

n . nnnn 

n.99Q  _9_99  _95 

0.0000 

0.0001 

0.0002 

0.0006 

0 . onna 

0.999  999  9 

0.0003 

0.0010 

0.0015 

0.0055 

n. nn77 

0.999  999  5 

0.0016 

0.0052 

0.0078 

0.0283 

0.0398 

0.999  999 

0,0021 

0.0071 

0.0106 

* ' 0.0384 

0.0538 

0.999  99*5 

0.007B 

0.0259 

0.0385 

0.1368 

0.1881 

0.999  99 

0.0084 

0.0280 

0.0415 

0.1445 

0.1939 

0.999  9 

0,0547 

0.1749 

0.2477 

0.6026 

0.5154 

- 0.999 

0.3238 

0.7219 

0.6519 

0. 0433 

0.0004 

0.99 

0.6008 

0.0359 

o.fl/inj 

0.0000 

0.0000 

0.9 

0.0004 

0.0000 

0.0000 

o.ooon 

0.0000 

0.8 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.7 

0,0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.6 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

£LuS 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.4 

0.0000 

0.0000 

n . noon 

n.nnnn 

n . nnnn 

0.3 

0.0000 

0.0000 

n . oooo 

0.0000 

0.0000 

0.2 

0.0000 

0.0000 

0.0000 

n nnnn 

or.  oooo 

0.1 

. o.nooo 

0.0000 

0.0000 

0.0000 

0.0000 

0.0 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

Table  A, 6 Bayesian  Calculation  Results  for  Various  Numbers  of  Tests, 

Each  with  One  Failure,  for  Binomial  Type  Prior- 


no 

x=- 

vo 


Tests  With  Uniform  Prior  Distribution 


; cell  Reliability 
Values , Ai 

99S  +1F 
or  IF  + 99S 

499S  + IF 
or  IF  + 499S - 

999S  + IF 
or  IF  + 999S  c 

4999S  + IF 
)r  IF  + 4999S 

9999S  + IF 
<2I-.11L+  ?.ms_ 

0.999  999  995 

0.0001 

0.0003 

0.0004 

0.0007 

0.0007 

0.999  999  99 

0.0002 

0.0007 

■HHM 

0.0014 

...  0.0015 

0.999  999  95 

0.0009 

0.0035 

0.0044 

0.0068 

n nm-* 

n 090  999  q 

0.0019 

0.0069 

0.0088 

' 0.0136 

_G._999  999  5 

0.0093  * 

0.0346 

1 

0.0681 

1 ! 

1 0.999  999 

0.0186 

0.0691 

0.0880 

■■  0.1357 

0.1436 

] 

0.999  995 

0.0928 

0.3448 

0.4381 

0.6652 

0.6900 

0.999  99 

0.0024 

0.0088 

....  0.0112 

0.0167  1 

0.0169 

0.999  9 

0,0184 

0.0658 

0.0797 

0.0827 

0.0534 

0.999 

•0.1682 

0.4196 

0.3241 

0.0092 

0.0001 

0.99 

0.6867 

0.0459 

0.0004 

0.  nrtnn 

0.0000 

0.9 

0.0005 

0.0000 

0.0000 

j 

0.B 

0.0000 

— 

0-7 

0.0000 

0.6 

o-onon 

0.5 

0.0000 

0.4 

0.0000 

! 0-5 

0.0000 

0.2 

0.0000 

0.1 

0.0000 

r 

r 

0.0 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

Table  Bayesian  Calculation  Results  for  Various  Numbers 

of  Tests#  Each  with  One  Failure,  for  Uniform  Prior. 
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Cell  Reliability 
Values,  A£ 

0,999  999  995 
0.999  999  99 
0.999999  95 


Tests  With  Peaked  {at  4th  Cell)  Prior  Distribution 


999S  + IP 


99S  + IF 
or  IF  + 99S 


0.0000 


0.0000 


0.0000 


0.0100 


0.0000 


0.0000 


0.0001 


0.0000 


0.0000 


0.0000 


0.0000 


0.0000 


4999S  + IF 
or  IF  + 4999S 

9999S  + IF 
or  IF  + 9999S 

0.0000 

0.0001 

0.0001 

0.0001 

0.0004 

0,0005 

0.0006 

0.0010 

0.0038 

0.0012 

0.0019 

0.0076 

0.0060 

0.0100 

0.0371 

0,0119 

0.0189 

0.0724 

0.1142 

0.1729 

0.4619 

0.7287 

0.7032 

0.0512 

0.0100 


0.0475 


Bayesian  Calculation  Results  for  Various  Numbers  of  Tests, 
Each  with  One  Failure,  for  Peaked  (at  4th  Cell)  Prior. 


Table  A. 8 
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Tests  With  Approximate  Shewed  Binomial  Prior  Distribution 
■ 1 1 — 1 ■ ■ » 


Cell  Reliability 
Values 

99S  +1F 
or  IF  +99S 

499S  + IF 
or  IF  +499S 

999S+1F 
or  1F+999S 

4999S+1F 
or  1F+4999S 

9999S  +1F 
or  1F+9999S 

fl-Qqq  300  OQC 

0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

0.999  999  99 

0.0000 

o.nnnn 

0.0000 

0.0000 

0.0000 

0.999  999  95 

0,0004 

0.0019 

0.0030 

0.0121 

0.0173 

0.999  999  9 

0.0003 

0.0015 

0.0024 

0.0096 

0.0137 

0.999  999  5 

0.0009 

0.0040 

n nnfi? 

0.0251 

0.0358 

0.999  999 

0.0012 

0.0053 

0.0082 

0.0330 

0.0470 

0.999  995 

0.0041 

0.0189 

0.0291 

0.1153 

0.1609 

n.qqq  qq 

0.0053 

0.0240 

0.0370 

0.1436 

0.1954 

0.999  9 

0.0348 

0.1531 

0.2255 

0.6105 

0,5300 

n qqq 

0 . 2385 

0.7325 

0.6880 

0.0508 

0.0005 

■ 0.99 

__  o.7i  an 

0.0587 

0.0006 

0.0000 

0.0000 

0.9 

0.0005 

0.0000 

0.0000 

i 

' 0.8 

i 

0.0000 

j 

0.7 

J 

0.6 

0.5 

- 0 4 

(V3 

£l*i 

oa 

r 

0.0 

1 0.0000 

0.0000 

0.0000 

0.0000 

0.0000 

Table  A.10  Bayesian  Calculation  Results  for  Various  Numbers  of  Tests, 

Each  with  One  Failure,  for  Skewed  Binomial  Prior. 
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TESTS  WITH  HO  FAILURES  AND  BINOMIAL  TYPE  PRIOR  DISTRIBUTION 
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TABLE  A»ll  ' BAYESIAN  CALCULATION  RESULTS  FOR  IOOOO  TESTS  WITH 
NO  FAILURES  s FOR  BINOMIAL  TYPE  PRIOR  DISTRIBUTION 


TESTS  WITH  NO  FAILURES  AMD  UNIFORM  PRICE  DISTRIBUTION 


TESTS  WITH  NO  FAILURES  AND  SKEWED  BINOMIAL  PRIOR  DISTRIBUTION 


CELL  RELIABILITY 
VALUES , A j. 

ioooo  s ! 

C.999  999  99 5 

i 

0.0011  i 

0.999  999  99 

0.0011 

C.999  999  9> 

0.52U0 

0.999  999  9 

0.2073 

C.999  999  5 

0.1037 

C.o99  599  i 

0.071a 

0.999  99i 

C.Giio^ 

C.999  99 

0.0296 

0.999  9 

O.OOtiO  ^ 

0.999 

0.0000 

0.99 

j 

k 1 

. 

C.9 

6.3  — ~ ^ 

1 

1 

* 0.6  i 

! 

0.5 

1 

MiSyiiipl 

1 

0^  i 

! 0.0000  1 

1 

TABLE  A.13  > BAYESIAN  CALCULATION  RESULTS  FOR  10000  TESTS  WITH  NO 

FAILURES,  FOR  SKEWED  BINOMIAL  PR  ICR  DISTRIBUTION. 
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APPENDIX  B 


Sample  of  Bayesian  Calculation  Computer  Program  in  FORTRAN  IV 


These  calculations  are  for  10,000  tests  in  which  the 
first  nine  tests  and  last  test  were  failures  with  the  remain- 
ing all  successes,  or  10  failures  and  9990  successes. 


FORTRAN  16K  COMPILER  PPN  07-S25-0004-10-N 

DIMENSION  A(24) ,0(24) ,P(24) ,C(24) ,PR(24) #ri <24) 

DIMENSION  IND  (10000) 

REA0(5.5)N1,N2 

RE A0 ( 5 , 15  ) ( A ( l ) i I *1 f N1 ) 

RE AD ( 5 , 15 ) (PU)#I»liNl) 

RE AD ( 5 » 15  ) ( B < I ) i I 31 1 Nl ) 

5 F0RMAT( I 3 » 15) 

10  FORMAT (2513) 

15  FCRMAT(6D12,9) 

00  60  1*1, N2 
I ND ( I ) *1 

IF  (loKU;  INI)  (I)  - 0 
IF  (IoEQ.1 ) GO  TO  19 
IF< I ,20,2) IND<  I >«0 

1 F < I , EQ , 3 ) I NO  C I >«0 
I T < I ,E0.4) I NO  C I >«0 
IF( I ,E0.5) I NO  C l ) *0 
I F ( I , EQ . 6 ) I ND  C I ) *0 
I F < I ,60.7) I ND  C I >«0 
I F C I ,EQt8> I ND ( I )*0 
IF( I .60.9) I NO  C I )«0 
IF< I , E0 , N2 ) I NO ( I ) *0 

1 F ( I , NE , N2  ) GO  TO  25 
19  WR I T£ ( 6 » 20  ) 

20  FORMAT(lHl#10X,iHA,15X#4HP<  A)  #1'6X,6HP(B/A)  #16X,  10NP(0/A)P(  A)  i 
116X , 6HP ( A/B ) // ) 

25  SUM*0 , 0 

DO  30  J«1*N1 
C(  J ) < J)«P(  J) 

I r ( I ND ( I ) ,£0,1)  C ( J ) * A ( J ) «P ( J) 
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3 PI  SUM«SUM*C(J> 

00  50  J*1.N1 
D«Ct JJ/SUM 
IF< I ,EQ,1>  GO  TO  35 
I r < I , KE . N2>  GO  TO  47 
I F ( I NO ( I ) ,E0,  1)  CO  TO  45 
35  WR I TE ( 6 » 40 ) A ( J) , P < J ) . B ( J V, C ( J ) , 0 
40  F0RmaT<iX,D16,9,F16.6,D26,9.F26.6,F24,6) 

IF(|,NE,1>  CO  TO  47 
PR< J)»P< J) 

IF(P( J)  ,LT.  0,0001)  PR ( J ) *0 ,0001 
GO  TO  47 

45  WRITE  ( 6. 4CT>  A ! J > ,P  < J ) , A t J)  , C < J ) , 0 
47  P ( J ) *0 
F I < J) *0 

IF<D, IT. 0,0001)  F I ( J ) »0 ,0001 
50  CONTINUE 
60  CONTINUE 
WRITE (2,70) 

70  F0RMAT(47H  TYPE  A 1 FOR  PUNCHED  DATA.  OTHERWISE  TYPE  A 0 ) 

READ  < 2 1 71 > I OR 

71  FORMATUl) 


IFUOP.NE.l)  GO  TO  80 
WR I TE ( 1 , 75 ) (PRCJ), J*1,N1> 
WR I TE ( 1 , 75  ) ( F I ( J ) , jsl , N1 ) 
75  FORMATC10F8.5) 

80  CONTINUE 
STOP 
END 
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ROBUSTNESS  STUDIES  FOR  BAYESIAN  DEVELOPMENTS 


IN 

RELIABILITY 

CHRIS  P.  TSOKOS  AND  A.N.V.  RAO 

DEPARTMENT  OF  MATHEMATICS 
UNIVERSITY  OF  SOUTH  FLORIDA 
TAMPA , FLORIDA  33620 

ABSTRACT 

The  Weibull  probability  density  function  is  considered  as  a failure  model 
under  the  influence  of  a stochastic  scale  parameter,  Bayesian  estimates  of  the 
scale  parameter,  reliability  function  and  hazard  rate  are  given  for  the  conjugate 
prior  distribution.  A method  for  evaluating  the  robustness  of  the  conjugate 
prior  distribution  which  characterizes  the  behavior  of  the  parameter  is  presented. 
The  proposed  method  is  based  on  any  acceptable  level  of  the  average  mean  square 
error  of  the  Bayesian  reliability  estimate  under  the  conjugate  prior  distribution. 
The  analytical  procedure  utilizes  the  ratio  of  the  average  mean  square  error 
when  the  prior  probability  distribution  is  different  from  the  conjugate  prior 
(and  its  Bayes  reliability  estimate  is  used)  to  the  average  mean  square  error  when 
he  prior  is  assumed  to  be  the  conjugate  distribution  (and  again  the  Bayes  estimate 
of  the  reliability  function  is  used), 

A computer  simulation  to  investigate  the  robustness  of  the  conjugate  prior 
distribution  with  respect  to  six  other  prior  probability  distributions,  namely, 
beta,  Poisson,  inverted  gamma,  truncated  normal,  log-normal  and  extreme  value,  were 
employed.  These  results  indicate  that  there  is  a significant  variation  in  the 
average  mean  squared  error  even  when  the  priors  were  chosen  so  that  their  first  two 
moments  approximately  agreed  with  that  of  the  conjugate. 

This  research  was  supported  by  the  United  States  Air  Force,  Air 
Force  Office  of  Scientific  Research,  under  Grant  No.  AF0SR-7^-27H • 
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1.  INTRODUCTION 


It  has  recently  become  quite  evident  that  Bayesian  reliability  methods  are 
very  useful  techniques  in  the  reliability  analysis  and  intepretation  of  various 
complex  systems.  A Bayesian  analysis  implies  the  expolitation  of  a suitable 
prior  information  in  association  with  Bayes*  Theorem.  It  rests  on  the  notion 
that  a parameter  within  a failure  model  is  not  simply  an  unknown  fixed  quantity 
but  rather  a stochastic  variable  which  follows  some  probability  distribution. 

In  life  testing  problems,  it  is  quite  realistic  to  assume  that  a life  parameter 
is  stochastically  dynamic.  This  assertation  is  strongly  supported  by  the  fact 
that  the  complexity  of  electronic  or  structural  systems  is  likely  to  cause 
undetected  component  interactions  resulting  in  unpredictable  fluctuations  of 
the  life  parameter, 

Drake  [4]  and  Evans,  [5],  have  given  an  excellent  account  for  the  importance 
and  implementation  of  Bayesian  statistics  in  reliability  problems.  From  a 
practical  point  of  view,  Feducia,  [6],  has  given  some  constructive  arguments 
concerning  the  Bayesian  usefulness  to  reliability.  Finally,  Crelin,  [3], 
among  others  has  given  an  eloquent  presentation  of  the  basic  philosophy  and 
fruitfulness  of  the  Bayesian  approach  to  reliability. 

Bhattacharya,  [1],  studied  the  exponential  failure  model  under  the  assumption 
that  the  parameter  behaved  as  a stochastic  variable.  Soland,  [9],  suggested  a 
Bayesian  analysis  of  the  Weibull  failure  model.  Canavos  and  Tsokos,  [2],  developed 
a Bayesian  analysis  of  the  scale  and  shape  parameter  in  the  Weibull  failure  model 
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and  its  corresponding  reliability  function.  For  the  scale  parameter  and 
reliability  function,  Bayesian  estimates  were  obtained  for  various  characterizations 
of  the  stochastic  behavior  of  this  parameter.  Recently,  Papadopoulos  and 
Tsokos,  [8],  have  developed  the  theory  for  constructing  Bayesian  confidence 
bounds  for  the  random  scale  parameter  and  the  reliability  function  of  the 
Weibull  failure  model.  The  usefulness  of  the  analytical  findings  were 
illustrated  by  employing  a Monte  Carlo  simulation. 

In  the  above  studies  among  others,  the  most  "popular”  prior  probability 
distribution  to  characterize  the  stochastic  behavior  of  the  scale  parameter 
of  the  Weibull  failure  model  is  the  conjugate  distribution.  Perhaps  the  most 
important  reason  for  employing  such  a prior  probability  density  is  because 
the  Bayesian  estimates  of  the  scale  parameter  and  reliability  function  result 
in  an  analytical  tractable  form.  Thus,  this  popularity  encourages  the 
investigation  of  what  happens  if  the  prior  probability  distribution  is  different 
from  the  assigned  conjugate,  but  we  proceed  to  employ  the  Bayes  reliability 
estimator.  Also,  how  "good"  is  the  Bayes  estimate  under  different  probability 
distributions  that  characterize  the  random  behavior  of  the  scale  parameter. 

It  is  the  aim  of  the  present  study  to  investigate  the  significance  of  the 
above  questions.  We  shall  propose  to  utilize  the  ratio  of  the  average  mean  square 
errors  as  a measure  of  "goodness"  of  the  Bayes  reliability  estimator.  In  Section  2 
we  introduce  some  of  the  preliminary  concepts  essential  to  our  investigations. 

The  Bayes  estimates  of  the  Acale.  pcvumzteAA,  fuitiabltAXy  function  and  the  hazard 
natd  are  given  in  Section  3.  In  Section  4 we  propose  an  analytical  approach  to 
Investigate  the  robustness  of  the  conjugate  prior.  A similar  procedure 
to  study  the  behavior  of  the  Bayesian  estimate  of  the  hazard  rate  is  given 
in  Section  5.  In  Section  6 we  present  a brief  computer  simulation  in 
an  attempt  to  illustrate  the  usefulness  of  the  basic  analytical  developments. 

The  conclusions  and  a summary  of  our  findings  in  the  present  study  are 
given  in  Section  7. 
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2.  PRELIMINARY  CONCEPTS 


In  this  section  we  shall  summarize  some  known  results  of  the  W eXbtM 
failure  model  that  we  will  be  utilizing  in  fulfilling  the  aims  of  the  present 
study. 

Consider  the  Weibull  failure  model  with  the  time  to  failure  probability 
density  function  given  by 

f(t;a,0)  = a0ta  ^ exp{-0ta},  0 < t < °°  (2.1) 

0 < a,  0 

where  0 and  a are  the  4 cal&  and  Ahcipz  parameters,  respectively.  It  is  well 
known  that  the  AeZlabltity  function,  R(t;a,e)  and  the  kazaAd  Acut&,  p(t;a»0) 
of  the  above  failure  model  are  given  by 


R(t;a,0)  = exp{-0ta} 

(2.2) 

o-l 

p(t;a,8)  =*  a0t  , 

(2.3) 

respectively. 

Suppose  now  that  the  scale  parameter  0 behaves  as  a AtockaAtic  vasUabZd 
and  that  the  shape  parameter  a is  known  or  can  be  estimated.  To  estimate  the 
reliabilty  function  under  the  described  situation  we  conduct  a life 

test  on  n items  and  observe  the  ordered  times  to  failure  t (<  n)  items  to 

— * 

be  tj , t2,  t3,  ...»  t . Let 

- = (tl’  t2*  ***>  V*  (2.4) 
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Then  the  likelihood  function,  L(_t;a,0),  of  the  complete  sample  is  given  by 

, r „i 

L(t;a,6)  = ~r  n‘~  s-t  (a6)r  II  t^  exp{-9T  } ' 

(n  - r).  1=1  i r 


T = I t?  + (n  - r)ta 
r i=l  1 r 


It  is  known,  [ 2 ],  that  the  probability  density  function  of  is  of 


fr(y|e)  = ^ z Yy~t  0r  yr  1 exp{~9y},  0 < y < ■”  . 
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3.  GAMMA  PRIOR  VlSTtUBUnON  FOR  THE  SCALE  PARAMETER 

Let  us  assume  that  the  stochastic  scale  parameter,  0,  of  the  Weibull 
failure  model,  (2.1),  is  being  characterized  by  the  natuAat  conjugate,  distribution, 
that  is,  the  gamma*  probability  density  function  given  by 

v 

0V  exp  0 < 0 < 

g(0)  = { 

0,  elsewhere  0 < y,  v 

Then  the  poAtcatoa  probability  density  function  of  0 given  t is 


gl(e;«|t)  - e-<r  + - + 1)  exp(-|(Tr  + v)> 

^OO 

,-(r  l x.+l)exp{"  6 (Tr  + U)  }de 


(Tr  + M)  T(r  + v) 


T + y I+V+1 

{■^s— } 


where  is  defined  by  equation  (2.6) 


*Note  if  the  failure  model  (2.1)  had  been  given  in  the  form  of 

a 


f(t;a,Q)  ^ ta  exp  {-  . 

then  the  conjugate  prior  distribution 

£>v+1 

g(e)  - exp  {-  f-1 


■ , 0 < t < 00 

0 < a,  0 

will  be  the  inverted  gamma,  that  is, 

0 . < 0 < * 

0 < y,  v 
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For  G to  be  characterized  by  the  natural  conjugate  prior  and  for  a 
Aquasit  nnxox  I066  function,  it  is  known,  [’2],  that  the  Bay&> 

0^(t),  of  the  scale  parameter  0 is  given  by  the  conditional  expectation 
of  0 given  t_.  That  is. 

Thy 

0 (t)  = E{ 0 1 1}  * rr-  , 1 < r + v . (3.3) 

c — r + v — t 

Also  under  the  above  conditions  the  B&yQ.A  AntJjlbiZAXy  QJ>tJjnaX,oA^  R^(t), 
is  given  by  the  conditional  expectation  of  the  reliability  function,  (2.2), 
given  t^,  (2.4).  It  can  be  easily  calculated  to  be 

R (t)  = E{R (t ; ot | 9)  1 1) 


(1  + 


ta 

T + y 

r 


r+v 


(3.4) 


Similarly,  the  Bat/eA  hazaAci  Aatz  eAtimcitZ.,  pc(t),  is  given  by 

p (t)  » E{p  ( t ; a | 0)  1 1 } 
c — 

- at0*  ^(r  h v)  , 

(Tr  + y) 


(3.5) 


In  the  following  sections  of  the  present  study  we  shall  employ  the 
above  Bayesian  estimates  to  investigate  the  robustness  of  the  Bayesian  reliability 
estimate  as  we  deviate  from  the  assumption  that  the  scale  parameter  is  being 
characterized  by  the  gamma  distribution. 
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4 . ANALYTICAL  APPROACH  TO  ROBUSTNESS 

In  the  preceding  sections  we  have  obtained  estimates  for  the  Bayesian 
reliability  and  hazard  rate  estimators  for  the  Weibull  failure  model  under 
the  assumption  that  the  stochastic  behavior  of  the  scale  parameter  is  being 
characterized  by  the  conjugate  distribution.  Perhaps  one  of  the  main  reasons 
for  choosing  such  a prior  probability  density  is  because  it  is  analytically 
tractable.  Thus,  it  becomes  quite  significant  to  ask: 

(ij  What  ifi  the  prior  distribution  is  di^erent  firom 
the  attigned  natural  conjugate , but  Me  employ  the 

A 

Bayet  reliability  estimator,  Rc(t),  developed  tfor 
the  invented  gemma? 

A 

[it]  Hom  "good"  it  the  estimate  Rc(t)  under  a dififierent 
prior  probability  distribution  £or  the  tcale 
parameter? 

To  answer  this  question,  we  shall  propose  to  employ  the  ratio  of  the 
average  mean  square  errort  as  a measure  of  "goodness”  of  the  Bayes  reliability 
estimator,  R^(t).  That  is,  the  ratio  of  the  average  mean 

square  error  when  the  prior  is  different  from  the  conjugate  prior  (and  Rc(t) 
is  used  as  the  reliability  estimate)  to  the  average  mean  square  error  when  the 
prior  is  assumed  to  be  the  conjugate  (and  R^(t)  is  again  used  as  the  Bayesian 
reliability  estimate).  Thus,  it  is  clear  that  the  closer  this  ratio  is  to  one, 
the  more  robust  is  the  Bayesian  reliability  estimate,  R^(t). 

We  proceed  by  developing  an  expression  for  the  average  mean  square  error 
in  terms  of  the  moments  of  the  prior  probability  distribution  which  characterizes 
the  behavior  of  the  scale  parameter. 
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The  definition  of  the  average  mean  square  error,  M.S.E,,  is  given  by 
M.S.E.  = E0{Et[R(t;o,0)  - Rc(t)]2|e}. 

More  conveniently  the  M.S.E,  can  be  written  in  the  following  form 
M.S.E.  - EQ(E  [R2 (t; a, 6) | 0 ] } 

- 2 E0{R(t;o,e)Et[Rc(t)|8]> 

+ Eg{Et[R2(t)]|6}.  (4.1) 

We  proceed  by  working  with  the  term 
E0{R(t;a,0)Et[Rc(t)|0]} 

of  equation  (4.1).  Utilizing  equations  (2.2)  and  (2.7)  we  can  write  the 
above  expression  as 


E^{R(t;o,9)Et[Rc(t)|9]} 

“ %{R(t;a,8) 


r oo  r 
r 6r 


(Tr+U) 


,r-l 


0 r (r)  (T4v+t“)r4v  r 

r 


exp[-  }.  (4.2) 


Let  us  consider 

r 


the  integral 


#00 

n 


r*h) 


0 (yHi )] 

o ^ (jrlp  +t°  ) r+V 


fV  1 exp{-  d0 


(4.3) 


We  can  write  the  above  integral,  equation  (4.3),  as  follows 

/•oo  I-  r+v) 

e_._  (y+p)  r-1  r y, 

nF)  7TTFTT7  y exp{-  V de 

J0  (y+y+t  ) 


(y+y ) r+V 
0 r(r)  (y+y+ta) 


a, r+v  y1'1  exP{-  0}  d0 


+ 


f el  (y^)r^ 

J»  r<0 


(y+y+t  ) 


a.  r+v  yr_1  eXp{_  0-}  dS 


=1+11. 


(4.4) 


We  shall  show  that  for  any  e > 0,  we  can  find  an  N > 0 such  that 


Ee(R(t;a,0)  II}  < e 

under  the  assumption  that  the  probability  density  function  of  0 is  bounded. 
Thus,  let  the  probability  density  function  of  G be  g(6),  such  that  g(0)  < k, 
0 < 0 < Then 


E {R(t ;a,0)  11} 


exp{-0 t }g(0)d6 


r Qr  / , *r+v 

-y.-H yT~  exp{-0y}dy 

N 1 (y+y+tV 


< K 


JO 


(exp  -6ta}r(r,N0)d0 
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K^{1^ 


(N+tV 


(4.5) 


where  T(r,x)  is  the  inc.omplzt<l  gamma  function. 


r_  i 

exp{-t)  t dt. 


From  equation  (4.5)  it  is  clear  that  by  choosing  N sufficiently  large,  the 
right  hand  side  of  the  inequality  can  be  made  smaller  than  e.  In  fact  it  can 
be  shown  that  if 


N > K 


a 1 
e t — 

{1  - r(r)}  r 

, t“  l 

1 -{1  - %)}r 


(4.6) 


then 


E0{R(t;a,6)  II}  <_  e. 


Similarly,  it  can  be  shown  that  if  N is  chosen  to  satisfy  the  inequality  (4.6) 


E0{Et[E2(t)|9]}  < 


fNR2(t)  il_  T r_1 

Rc(t)  T(r)  r 

0 


e dT  + £ 
r 


(4.7) 


, then 
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where  y is  the  nth  ordinary  (noncentral)  moment  of  the  prior  probability 
distribution  function  of  the  stochastic  variate  0,  g(0). 

Thus,  inequality  (4.9)  gives  an  approximate  value  for  the  average  mean 
square  error  in  terms  of  the  moments  of  the  prior  probability  distribution. 


AueAflge  Mean  Square  E/laosi  Post  the  Conjugate 
Vi^pilbiution 


Using  the  definition  of  M.S.E.,  equation  (4.1),  the  reliability  function 
of  the  Weibull  failure  model,  equation  (2.2)  and  the  conjugate  prior  distribution, 
equation  (3.1)  we  have 


Eg{R(t;a,6)  Efc [ Rc (c ) | 6 ] } 


Jo 


6^  exp[-y6]  exp[-0ta] 


t +y 


r(  v)  r(r) 


y T (r-fV)  r 


^ r ^r+u  Tr-1  r(r+v) 


o Tr+ma 


1 (Tr+y+t“)r+v 


rTrTFRT  yr  B(v»r)  2V2(t"*v)»  r » r-^i  ** 


v+r  2 «-a 

- w B(r,v)}z  2F1[2(r*+v)>  r;  r-fv ; ~ 


dT 

] 


Using  inequalities  (4.6)  and  (4,7)  in  equation  (4.1)  we  can  obtain  an 
approximation  to  the  average  mean  square  error,  that  is, 


M.S.E.  < E„  ( 


N rtr 


(T  -H.)2(r+V) 


0 n r(r) 


(Tr+u+t“) 


S72(?+v)  Tr  {exp  -V>dV 


fN  .r 

+ 2 Ee  {j  exp[-et°] 


r (T^+u ) 

. T exp{-T  0}dT  } 

r)  (TrW‘)r+V  r r r' 


- E.fexpl^et01]}  + 2c. 


(4.8) 


fNizi 

~ in  p 


k+r-1 


(r)  k! 


(rUJL.)Ur^»  dy 

y+u+t“ 


fN  , .k  , a .k+r-1 
b,  = iiiL-  i£  +y) ( -zjn 

k 0 r<r)  k-  1 ^ a 

U y+u+t 


\r+v  j 
) dy 


. (-D 


(2t“)k. 


Using  these  definitions,  inequality  (4.8)  can  be  written  as  follows 


M.S.E.  < l {A  - 2 B.  + C.  } p(r+k)  + 2 e 
k-0  K k k 


(4.9) 
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rl  - 

where  B(x,y)  = tX  1 (1  - t)y_1  dt 

Jo 


F(a,6;  V;  z) 


is  a hypergeotnetric  function.  For  some  details  see  [ 7,  p.  287]. 
Similarly, 


En{R  2(t)|0}  = -L_  1 


k! L_  r f V*  ,2(r-Ki)  r-1 

r(v)  r(r)  Jo 


{ 0r+v  1 exp  (p+T  )0]d0>dT 
Jn  r r 


= V r{B(v,r)}2  F [2(v+r),  r;  r+u;  - 


E {exp[-2ta0] } 


- exp[-2ta6]  ~yr~T  e“yG  d0 

Jo  r(v) 


JL_  r(v) 

r(v)  <p+2tV 


a 

(1  + 2 T-  )V 

u 
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Hence,  the  average  mean  square  error,  M.S.E.,  for  the  conjugate  prior 
distribution  is  given  by 


M.S.E  . = 
c 


a 

<1+2— )V 
U 


- yV+r{B(v,r)}2 


r;  r+v;  - —■  ] 


(4.9a) 


Thus,  as  we  have  mentioned  above,  a measure  of  the  robustness,  R(t),  is  the 
ratio  of  the  mean  square  error  when  the  prior  is  different  from  the  conjugate, 
M.S.E,,  to  the  average  mean  square  error  when  the  prior  is  assumed  to  be  the 
conjugate,  M.S.E^.  That  is, 

R(t)  = - (4,10) 

M.S.E  . 
c 

For  the  conjugate  prior  we  have 


R(t)  = 


M.S.E. 


(1+2  ~ ) ';-yV+r{B(v,r)}22F1[2(r+v) , r;  r+v,  - ~ ] 


Unfortunately,  the  M.S.E.  for  most  priors  other  than  the  conjugate  cannot  be 
given  in  a closed  mathematical  form.  Thus,  we  must  rely  on  approximating  the 
M.S.E.  for  other  prior  distributions  using  equation  (4.9)  in  conjunction  with 
electronic  computers. 
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5,  HAZARP  RATE  BEHAVIOR 


In  this  section  we  shall  propose  to  use  the  method  we  discussed  in  the 
previous  section  to  investigate  the  robustness  of  the  Bayesian  estimate  of  the 
hazard  rate  using  the  prior  conjugate  with  respect  to  other  prior  distributions. 
Namely,  we  shall  employ  the  ratio  of  the  average  mean  square  error  when  the 
prior  is  different  from  the  conjugate  prior  (and  Pc(t)  is  used  as  the  estimate 
of  the  hazard  rate)  to  the  average  mean  square  error  when  the  prior  is  assumed 
to  be  the  conjugate  (and  Pc(t)  is  again  used  as  the  Bayesian  hazard  rate  estimate). 
The  average  mean  square  error,  M.S.E.^  for  the  hazard  rate  is  given  by 

M.S.E.j,  = E0{Et[p(t;«,e)  - Pc(t)]2|e) 

■ E0{p(t;a,9)}2  - 2E0{p(t;a,6)Et[pc(t)|8]} 

+ E0{Et[pc(t)]2|0}.  (5.1) 


Recall  that  the  Bayes  estimate  of  the  hazard  rate  is  given  by 


P (t) 


atn~^  (r-H>) 

(Tr+u) 


and  that  follows  the  gamma  distribution,  (2.7),  Following  the  procedure  w*  h«te 
employed  in  the  previous  section  we  can  calculate  the  average  mean  square  error 
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for  the  Bayes  estimate  of  the  hazard  rate.  That  is,  from  equation  (5.1)  we  have 


M.S.E., 


a2  2(«-l),„,„2 


(E(QZ)  - 2 l C E(0k+r+1) 
k=0 


+ l D E(0k)} 
k=0 


(5.2) 


where 


(-1) 


k (r+v) 


k 


(r-l)I  kl  (y+y ) 


dy 


and 


D, 


N (-l)k 


k 


0 (r-1):  k:  (y+u)2 


dy. 


Thus,  for  a specified  (very)  small  z we  can  obtain  an  N that  satisfies  inequality 


(4,6)  and  one  can  proceed  to  calculate  the  H.S.E.  for  a specified  probability 

ll 

distribution  for  the  parameter  0 using  equation  (5.2). 

AveAage  Mean  SquaAe  Viaoa  Oi j the  Hazcuid 
Rate  Vo  A the  Conjugate  VtitsUbutton 

The  average  mean  square  error  for  the  Bayesian  hazard  rate  estimate  for  the 


conjugate  prior,  M.S.E,  ; can  be  obtained  to  be 

H 


■7 -—y  2 2 (a-1) . a(a+l)  (r+v)2  B(r.  v+2) 

m's,e-h  “ s { — 2 20+T)  ¥(7tv5 


(5.3) 


where  B(x,y)  is  the  beta  function  defined  in  Section  4. 


Thus,  one  can  employ  a procedure  similar  to  the  one  we  discussed  previously 

to  study  the  robustness  of  the  Bayesian  hazard  rate  of  a conjugate  prior  with 

respect  to  other  prior  probability  distributions  by  forming  the  ratio  of 

M.S.E, H to  M.S.E.  . 

c 


6.  COMPUTER  SIMULATION 


As  we  have  pointed  out  in  the  previous  sections  to  investigate  the 
robustness  of  the  natural  conjugate  with  respect  to  different  prior 
probability  distributions  that  characterize  the  behavior  of  the  scale 
parameter  we  propose  to  study  the  ratio  of  the  average  mean  square  error 
with  a non-conjugate  prior,  M.S.E.  to  the  average  mean  square  error  using 
a conjugate  prior,  M.S.E.^.  That  is,  the  measure  of  robustness,  R(t-)  is 
given  by 


R(t) 


M.S.E. 


M.S.E. 


Our  sensitivity  analysis  will  be  determined  by  a careful  computer 
simulation.  A brief  description  of  the  simulation  procedure  is  given  below: 
i)  A very  small  £ is  specified  and  an  N is  obtained  which  satisfies 
inequality  (4.6). 

ii)  For  specific  values  of  the  parameters  and  the  conjugate  prior  the 
is  calculated  using  equation  (4.9a). 

ill)  For  the  different  prior  probability  distributions  given  in  Table  6.1 
the  M.S.E.  is  calculated  using  equation  (4.9). 
iv)  The  measure  of  robustness,  R(t)  is  calculated  and  plotted  as  a 
function  of  time  for  different  configurations  of  the  parameters. 
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Table  6.1  gives  the  different  prior  distributions  along  with  the  various  values 

of  the  parameters  that  we  have  used  in  the  simulation.  These  values  are  chosen 

so  that  the  mean  and  variance  of  the  specified  prior  is  approximately 

close  to  the  mean  and  variance  of  the  natural  conjugate  prior.  For  the 

1 

Weibull  failure  model  the  shape  parameter  was  fixed  at  a - y,  1,  2.  The 
parameters  of  the  natural  conjugate  were  p - 60  and  v ” 6. 

Figures  6.1  - 6,6  are  representative  samples  of  the  proposed  measure  of 
robustness,  R(t),  of  the  conjugate  prior  distribution,  with  respect  to  the 
following  different  priors;  beta,  PotAACn,  invented  gamma. , truncated  noftmaZ, 
tog -no mat  and  the  extreme  value.. 
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7.  SUMMARY  ANV  CONCLUSIONS 


Bayesian  estimates  of  the  scale  parameter  and  reliability  function  of  the 
Weibull  failure  model  under  the  "popular"  conjugate  prior  have  been  given. 

The  employment  of  this  prior  results  in  analytically  tractable  forms  of  the 
Bayesian  estimates  is  of  interest,  A procedure  has  been  developed  to  investigate 
the  consequence  of  the  prior  probability  distribution  being  different  from  the 
assigned  conjugate.  The  proposed  method  employs  the  ratio  of  the  average  mean 
square  error  when  the  prior  is  different  from  the  conjugate  prior  (and  its  Bayes 
reliability  estimate  is  used)  to  the  average  mean  square  error  when  the  prior  is 
assumed  to  be  the  conjugate  distribution  (and  again  the  Bayes  reliability  estimate 
is  used) . That  is,  we  shall  assume  that  the  level  of  the  average  mean  square 
error  of  the  Bayesian  reliability  estimate  under  the  conjugate  prior  (which  is 
assumed  to  be  the  true  prior)  is  acceptable.  The  present  study  investigates 
M.S.E.  under  different  priors  and  compares  them  to  the  accepted 

The  proposed  measure  of  "goodness"  of  the  Bayes  reliability  estimate  under 
the  conjugate  prior  is  the  ratio 

E {E  [R(t;a,0)  - R_(t) ]2 | 0} 

R(t)  - 1 L ' 

E0{Et[R(t;a,8)  - Rc(t)]2|e)c 

The  closer  this  ratio  is  to  one,  the  more  robust  is  the  conjugate. distribution  as  a 
prior  to  obtain  a Bayesian  reliability  estimate.  Thus,  there  are  three  significant 
possibilities  the  proposed  ratio  as  a function  of  time  will  assume. 


(i)  The  ratio  will  be  approximately  equal  to  one,  in  which  cate  we 
can  conclude  that  the  conjugate  prior  distribution  is  quite 
robust.  Thus,  since  the  conjugate  prior  it  mathematically  mote 
attractive  to  work  with,  it*  implementation  .lb  recommended. 

(ii)  The  ratio  is  significantly  greater  than  one . In  this  case  the 
use  ot j the  conjugate  prior  u)ill  rebult  in  considerable  increase 
in  the  average  mean  square  error.  By  inspecting  the  above  ratio, 

R(t),  it  is  clear  that  the  numerator,  which  represents  the  average 
mean  square  error  o f the  Bayesian  reliability  estimate  under  the 
influence  of  the  conjugate  prior,  when  in  fact  the  true  prior  is 
different  from  the  conjugate,  is  considerably  larger  than  the 
denominator  which  represents  the  average  mean  square  of  the  Bayesian 
reliability  estimate  under  the  conjugate  prior  when  in  fact  the 
true  prior  is  the  conjugate.  Thus,  it  is  recommended  that  a more 
careful  choice  of  the  prior  distribution  be  investigated.  In  fact, 
any  non-conjugate  prior  which  results  in  R(t)  greater  than  one  would 
be  a better  candidate  for  characterization  of  the  stochastic  behavior 
of  the  parameter. 

(iii)  The  ratio  of  R(t)  is  significantly  less  than  one.  In  this  case  the 

use  of  the  conjugate  prior  is  not  objected  because  it  yields  an  average 
mean  square  error  which  is  less  than  the  accepted  level. 
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A similar  approach  can  be  employed  to  study  the  Bayesian  estimate  of  the 
hazard  rate  under  the  influence  of  the  conjugate  prior  versus  different  prior 
distributions. 

A brief  computer  simulation  has  been  given  utilizing  the  six  different 
prior  probability  density  functions,  namely,  b&ta,  Po^LA^on,  XnveAte.d  gamma , 
tJuw.CLCLt<id  nOAmat,  Zog~nOAinaZ  and  <LX&uimQ.  vatu<L.  The  computer  results  indicate 
that:  there  is  a significant  variation  in  the  M.S.E.  when  the  priors  were  chosen 
so  that  their  first  two  moments  approximately  agreed  with  that  of  the  conjugate. 
One  would  therefore  conclude  that  it  is  the  higher  moments  that  cause  the 
variation.  This  observation  is  in  agreement  with  the  analytical  expressions  for 
robustness  which  show  that  the  M.S.E.  depends  on  the  higher  order  moments.  Thus, 
it  appears  that  in  Bayesian  reliability  estimation  one  has  to  be  very  careful 
in  choosing  (estimating)  the  prior  probability  distribution  to  characterize  the 
stochastic  parameter  of  the  given  failure  model. 
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A BAYESIAN  APPROACH  TO  RELIABILITY  GROWTH  ANALYSIS 
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ABSTRACT : A Bayesian  approach  to  assessment  of  reliability 

for  systems  under-going  active  development  is  presented.  It  is 
assumed,  that  at  distinct  stages  of  development,  tests  are  conducted 
on  similar  Items  providing  data  for  estimation  of  reliability  and 
for  the  tracking  and  projection  of  reliability  growth.  While  a 
parametric  model  is  not  assumed,  it  is  assumed  that  reliability  is 
non-decreasing  from  stage  to  stage.  An  approximation  of  the  poste- 
rior conditional  distribution  of  the  nth  stage  reliability  given 
that  it  is  at  least  as  great  as  the  reliability  In  the  (n-l)th 
stage  is  employed.  Properties  of  estimates  extracted  from  this 
distribution  are  examined.  Illustrative  numerical  examples  are 
provided . 

INTRODUCTION : Reliability  has  for  some  time  been  regarded  as 

one  of  the  single  most  important  characteristics  by  which  adequacy 
of  system  design  can  be  measured.  Prior  to  the  initiation  of  the 
development  program  general  requirements  for  the  system  are  estab- 
lished. One  of  these  requirements  Is  often  the  minimum  reliability 
to  be  considered  acceptable  for  the  system  design  resulting  as  the 
end  product  of  the  development  effort.  An  important  facet  of  the 
development  program  is  the  systematic  effort  on  the  part  of  respon- 
sible engineers  to  see  that  these  reliability  goals  are  achieved. 

The  level  of  engineering  effort  is  determined  by  program  managers 
who  are  responsible  for  the  allocation  of  available  funds,  time, 
manpower,  and  facilities  during  development  to  insure  that  design 
goals  are  achieved.  Program  managers  have  a need  for  a consistent 
quantitative  technique  by  which  they  can  monitor  and  evaluate, 
through  assessment  of  current  status  and  forecasting  of  future 
trends,  the  progress  of  the  development  program  toward  attainment 
of  the  goals.  It  was  this  need  that  inspired  the  development  of 
the  concept  of  reliability  growth  analysis. 

The  phenomenon  of  reliability  growth,  as  it  occurs  in  Army 
materiel  development  programs,  does  not  occur  simply  as  a result 
of  placing  prototype  systems  on  test  with  reliability  growing  in 
proportion  to  the  time  or  number  of  tests  occurring  before  a failure 
is  observed.  The  Ingredients  of  a development  program  which  cause 
reliability  growth  are  many  and  complex.  First  and  foremost  are 
the,  hopefully  well-designed,  test  programs  which  together  become 
an  iterative  process  of  design,  test,  redesign,  retest,  and  so  on. 
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Following  each  test  stage  the  redesigns  are  accomplished  in  part 
for  the  purpose  of  eliminating  observed  failure  modes,  as  well  as 
any  potential  failure  modes  or  deficiencies  which  may  have  been 
recognized  since  the  last  design.  Many  reliability  growth  models 
appearing  in  the  literature  consider  only  this  process,  and  as  a 
result,  fail  to  accurately  describe  reliability  growth  in  the  con- 
text  of  a realistic  Army  development  program.  Some  of  the  other 
factors  which  influence  end  item  reliability  and  which  should  be 
taken  into  account  when  constructing  reliability  growth  models  are 
advancements  in  the  state-of-the-art  allowing  for  the  elimination 
of  deficiencies  considered  inherent  in  the  initial  design,  improved 
inspection  and  quality  control  procedures,  identification  of  and 
concentration  on  important  parameters,  better  methods  of  .incorpo- 
rating design  changes,  accelerated  or  overstress  testing  in  combina- 
tion with  better  understanding  of  the  environmental  stresses  the 
system  will  experience  in  actual  use,  and  many  more. 

That  these1  factors  and  their  interactions  exist  allows  for  the 
following  important  preliminary  conclusions: 

1.  Reliability  Is  increasing  from  stage  to  stage  of  testing. 

2.  A general  representation  of  reliability  growth  by  a smooth 
curve  is  unreasonable. 

The  first  conclusion  is  reached  because  the  influencing  factors 
provide  a "built-in”  guarantee  that  the  resultant  system  will  be 
improved;  the  second,  because  the  impact  of  interacting  factors 
between  stages  will  vary  significantly. 

The  approach  to  be  presented  in  this  paper  incorporates  the 
first  conclusion  as  an  underlying  assumption  and  the  second  con- 
clusion through  avoidance  of  parametric  models  [2]  representing 
reliability  as  a function  of  development  program  time.  The  methods 
of  estimating  reliability  at  each  stage  are  based  on  the  given 
condition  that  the  reliability  has  not  decreased  since  the  last 
stage.  At  each  test  stage  similar  items  are  placed  on  test 
providing  binomial  data  in  the  form  of  successes  and  failures 
from  which  estimates  of  the  current  value  of  reliability  are  to 
be  made.  The  unknown  reliability  at  each  stage  is  treated  as  a 
random  variable  which  is  the  unknown  parameter  of  the  Bernoulli 
process  generating  the  data  in  that  stage.  Prior  and  posterior 
distributions  are  formulated  as  beta  distributions,  the  family 
of  distributions  conjugate  to  the  Bernoulli  process  [1].  It  will 
be  apparent  that  conjugate  priors  are  not  requisite  to  application 
of  the  approach*  In  actual  applications  one  could  use  person- 
alized or  evidential  priors  following  essentially  the  same  steps 
outlined  herein.  The  prior  distributions,  whether  they  be  conjugate 
or  otherwise,  and  the  likelihood  functions  based  on  the  observed  data 
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in  each  stage  are  used  to  compute,  via  Bayes  Theorem,  the  posterior 
conditional  distribution  of  the  nth  stage  reliability  given  that  it 
is  greater  than  or  equal  to  the  reliability  at  the  (n-l)th  stage. 
This  distribution  can  then  be  used  to  make  inferences  about  the 
nth  stage  reliability  in  the  context  of  reliability  growth.  The 
details  of  the  approach  are  provided  in  the  next  section. 


PROCEDURE  FOR  TRACKING INDEPENDENT  BERNOULLI  PROCESSES: 

The  testing  during  the  development  program  Is  conducted  in  m distinct 
stages.  At  the  1th  stage  the  results  of  testing  are  recorded  as  the 
observed  number,  , of  failures  out  of  trials.  We  denote  by 


st=  - x^.s  the  number  of  successes  observed  at  the  ith  stage.  The 

ith  stage  reliability,  that  is,  probability  of  success,  is  denoted 
by  r^.  The  factors  listed  previously.  Including  elimination  of 

failure  modes  observed  in  previous  stages,  lead  to  improvements  in 
system  reliability  from  stage  to  stage  so  that  we  may  assume  the 
r^fs  are  nondecreasing,  or  equivalently,  r^_  <,  r2  4 . . . £rm.  In 

utilizing  the  Bayesian  approach  we  will  consider  the  unknown  reli- 
abilities as  independent  random  variables.  When  it  becomes  neces- 
sary to  emphasize  that  the  unknown  ith  stage  reliability  Is  a random 
variable  In  the  Bayesian . sense , it  will  be  denoted  by  r^.  The 

likelihood  for  the  ith  stage  event  is  given  by 

LCr^s^np  = (£i)  r1si(l  - r1)ni"si  ( 

To  apply  Bayes  Theorem  for  the  determination  of  the  posterior 
probability  density  functions  of  the  r^  we  require  formulation  of 

prior  distributions  for  these  random  variables.  As  stated  previously 
the  conjugate  beta  family  will  provide  the  class  of  distributions 
from  which  a choice  will  be  made.  Our  studies  have  shown  that  good 
estimation  procedures  in  the  reliability  growth  environment  result 
when  a uniform  distribution  on  the  unit  interval  is  assumed  as  the 
prior  distribution  for  each  r*j_.  As  members  of  the  beta  family  the 


densities  of  these  prior  distributions  are  formulated  as 


r (a0i+b0i) 

i i , m 

r<aoi)r<bol) 


0 < rj<  1 

i=l, , . . ,m 
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where  the  parameters  of  the  beta  distributions  are  a0^  =boi  -1» 

1=1,.,., m,  with  the  o subscript  and  prime  used  to  signify  that  these 
are  prior  representations.  To  employ  other  members  of  the  beta 
family  (i.e,  non-uniform)  as  priors  all  that  is  required  is  a 
redefinition  of  the  aQ^ f s and  bQlTs.  It  is  suggested,  however,  that 

the  impact  on  the  properties  of  resulting  estimates  be  thoroughly 
understood  beforehand.  Studies  of  prior, representations  for  the 
reliability  growth  situation  are  now  being  undertaken. 

■Employing  Bayes  Theorem  following  each  stage  of  testing,  we 
determine  the  unconditional  posterior  distribution  of  the  r^  to  have 
density 



1 flt~  k (ri)L(ri;si,ni)dri 

0 i 


r (ni  + + bQ ) 

r(si+aoi>r<ni-si+boi) 


s -+a 


r-:  i 


oi 


(l-r±) 


m -Si +b 


oi 


Letting  aj_  = Sj_  + and  = n^  - s1  + bQi , we  can  reformulate 

this  density  as 


f"r.(ri) 


rUj  + bj) 
TtatfTib}) 


rfi"1(l-ri)bi“1 


(1) 


Having  the  probability  densities  for  the  independent  random 
variables,  r^ , we  now  wish  to  structure  our  procedure  for  assessing 

reliability  at  each  stage  within  the  context  of  reliability  growth. 
We  do  this  by  taking  advantage  of  our  primary  assumption,  r^  < r ^ 

...  4 ifo  , considering  at  the  kth  stage  that  4 r^  4 * * * < add 

then  introducing  the  additional  assumption  that  i^-l  £ \ * The 

manner  in  which  this  is  accomplished  is  by  first  formulating  the 
unconditional  joint  posterior  density  of  r^  and  as 


^rl'r2*  f "r2  ^2^  ' 0 = rl'r2=^  1 - 
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To  simplify  notation  we  will  for  the  remainder  of  this  discussion 
drop  the  subscript  of  the  density  functions  indicating  the  appropriate 
random  variables  unless  ambiguity  will  result.  Thus  ^lrr2^x*^ 

will  be  simply  represented  as 


h[xltx2)  = f"(r1)f"(r2)  . (2) 

The  conditional  joint  posterior  distribution  of  r1  and  r2  given 
tl  < r2  then  has  density 

h(rlfr2) 

h(rrr2|ri<  r2)  

i/h(rl'r2)drldr2 

for  (r19r2)eS,  where  S=  { (ri,r2):  0.1  rl  1 r2  1 U* 

Hence,  from  (2) 


Mr^ralr^  r2)  = 

/ljr2f»(r1)f»(r2)dr1dr2 
o o 

= 0 ; elsewhere  , 


; 0 r2i  1 


To  obtain-  the  conditional  marginal  posterior  density  of  r2 
given  1^2  we  integrate  with  respect  to  ri  over  its  range.  Hence, 

h(r2|rilr2)  * [f" (r2) /^2f " (rx) drx] / " (rx) f1 " (r2) drxdr2]  (3) 

The  denominator  of  the  right  side  of  (3)  is  the  posterior  probability 
that  H < r2  , Substituting  the  unconditional  marginal  distributions 

given  by  (1)  in  (3)  and  cancelling  the  normalizing  constants  we  have 


Mralr^j)  = ± l 2 i L - 

b2-lr|l“l  (1-^)  bi-ldr;Ldr2 
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To  remain  consistent  with  our  choice  of  the  beta  family  to 
summarize  beliefs  concerning  an  unknown  reliability  and  to  eliminate 
the  difficulties  expected  in  using  the  form  of  the  density  appearing 
in  (4)  we  will  approximate  this  density  by  a beta  density.  The  beta 
fit  will  be  accomplished  through  use  of  the  method  of  moments  employ- 
ing only  the  first  two  moments  of  the  distribution  with  density 
given  by  (4)  to  obtain  the  two  parameters  of  the  desired  beta 
density.  If  we  let  p12  and  y22  denote  the  first  and  second  moments, 

respectively,  of  this  distribution,  we  can  compute  their  values  from 


y12 


= f1  r. 


h(r. 


l=r2 


) dr- 


(5) 


and 


(6) 


The  required  double  Integrations  are  performed  using  the  binomial 
expansion  for  the  interior  integrals,  in  this  case  the  incomplete 
beta  integrals,  and  summations  of  the  complete  beta  integrals 
forming  a finite  or  convergent  infinite  series. 

To  determine  the  beta  distribution  with  first  moment  u12  and 
second  moment  y22  we  recall  that  a beta  distributed  random  variable 
y with  density 

f (y)  . £k.t.S>  yP-^l-yj'J-1  ; 0 < y < 1 

6 r (p)r  (q)  ~ - 


has  n th  moment  given  by 

E(  yn  ) = t n [ (p+i-X) / (p+q+i-l) ] , 
1=0 


Hence,  two  equations  can  be  solved  for  p and  q , the  parameters 

2 2 

of  the  beta  fit  to  h(r2  1^  <r2  ) given  in  (4);  these  are 

^12  53  p2  / ( + q2  ^ (7) 
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and 


p22  = [ p2  ( p2  + 1 ) l/[  (p2  + q2)  <p2  + q2  + i)  I 


(8) 


The  solutions  for  p and q are 

*2  2 


P2  = [u12(1-,,12>/  a2]  ~ ?12 


(9) 


and 


[Ul2(1-1Ji2,Z/a|]  + H12  - 1 


(10) 


where  a^=  y2-  p^2  is  the  variance  of  the  distribution  with  density 
given  by  (4).  We  will  represent  by  g2(r2)  the  beta  density 
approximating  h(r2|r1:<  r2  ) . Hence, 

g,(r2)  = r(p2  I M ^2-l(1.r2)q2-l 

r (p2)  r (q2) 

where  p2  and  are  given  by  (9)  and  (10),  respectively.  Obviously, 
the  beta  distribution  summarizing  our  beliefs  concerning  has 
density  given  by  f"^)  so  that  we  can  define  gx  (rx)  by 

g1(r1)  * f" (ri) # 

Point  and  Bayesian  confidence  interval  estimates  of  r^  can 
be  obtained  using  g Similarly,  these  estimates  of  r can  be 
obtained  using  g2 . The  point  estimates  of  r ± and  r2  are  the  means 
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of  the  distributions  with  densities  approximated  by  g ]_  and  g 2, 
respectively.  Hence, 

- ax  . - P2 

rn= and  r2- . 

al  + bl  P2  + <32 

Lower  100(1  - y )%  Bayesian  confidence  limits  for  rx  and  r2  are 
obtained  from  solution  for  of 

Y = / gi(zi)dzi  i - 1,2  . 

o 

To  track  reliability  growth  through  the  remaining  stages 
we  require  the  conditional  marginal  posterior  distributions  of  the 
k=3,...,m.  Instead  we  will  obtain  the  beta  approximations  to 

the  densities  of  these  distributions  as  we  did  for  the  conditional 
marginal  posterior  density  of  r2.  for  k = 3,...,  m we  begin  with 

the  conditional  joint  posterior  distribution  of  rK_x  and  rk  given 
rl  ~ ^2=^  9 • * • , ^ The  density  of  this  distribution  is  approx- 

imated by 


htrk-i'V 


ri* 


r-vS 

£=r=. 


f" (rk)gk-l(rk”l) 


The  conditional  joint  posterior  distribution  of  and  rk  with 

the  additional  given  condition  that  rj^  £ fk  is  then  approximated  by 


h(rk<-l'rkl*i  <•••  <*k>  = 


f" (rk)gk-l(rk-l) 


f fkf " (rk>  9k-l  (rk-l>  drk-ldrJ 
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Therefore,  the  conditional  marginal  posterior  density  of  rk 
given 


h(rk  | r^,  , *<Xy-) 


f " <rk>  -fo^k-l  <rk-l> drk-l 

1 1 ■*  ■* — <* 

lYkf"  (rk)  ‘Jk-l  <rk-l)  ^k-idrk 


(11) 


To  determine  gk(rk),  the  beta  fit  to  (11),  we  again  employ  the 

method  of  moments.  The  first  and  second  moments  of  the  dis- 
tribution with  approximate  density  given  by  (11)  are,  respectively, 

f1  I -V 

uik  = / rkh(rklrl^--^rk)drk 


and 


2k 


= l rkh(rkiJii---^k)dri 


Proceeding  as  we  did  for  k=2,  the  values  of  and  v<2kcan 
be  used  to  determine  the  parameters,  and  qk,  of  gk  from 

Pk  ” 1 uik(l~ulk)/oj;]  -Uik  (12) 


and 

3 k = [ tikU”Pik^akJ  +yik“  1 * 


(13) 
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So  that  the  conditional  marginal  posterior  density  of  rk  is 
approximated  by  the  beta  density 


9k<rk> 


HPk  + qk) 
r(pk)r(qk) 


rkk'1<i-rk) 


q^1 


and  the  estimate  of  is  the  mean  of  gk ; that  is. 


*k  = Pk  / (Pk  + Gk>  . (14) 

The  lower  100(1  -y)%  Bayesian  confidence  limit  of  r k is  r 
determined  from 

y = l 9k(rk,drk  • (15) 

We  can  now  employ  (14)  and  (15)  to  track  reliability  growth 
through  each  stage  of  the  development  program  testing. 

BX AMP LK  OF  APPLICATION:  The  results  in  TABLE  1 represent  the 

data  recorded  for  seven  stages  of  testing  during  a hypothetical 
development  program. 


TABLE  1 


STAGE,  i 
SUCCESSES,  s1 


4 5 6 7 

6898 


FAILURES,  x ± 4534212 

TESTS,  n.  10  10  10  10  10  10  10 

Suppose  we  are  confident  that  the  factors  we  listed  earlier 
which  influence  reliability  growth  are  in  effect  so  that  we, 
without  question,  assume  that  test  items  at  each  stage  are 
improved  over  items  tested  in  previous  stages  and,  as  a conse- 
quence, reliability  Is  nondecreasing  from  stage-to— stage . We 
assume  that  the  items  are  homogeneous  within  each  stage. 
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Before  proceeding  with  our  analysis  of  this  data,  we  must 
emphasize  here  that  at  all  times  our  assumption  of  non-decreasing 
reliability  should  rest,  not  only  on  the  subjective  judgment  that 
the  influencing  factors  are  obtaining,  but  also  on  careful 
consideration  of  the  data.  If,  for  example,  the  data  up  to 
the  kth  stage  yields  a very  small  posterior  probability  that 
rk;>  rk_i  we  would  seriously  question  our  assumption  and  treat 

the  data  in  a manner  different  from  that  provided  In  the  preceding 
section.  It  is  Important  that  this  is  kept  in  mind  when  using 
this  approach. 

The  data  in  Table  1 was  used  to  track  reliability  through 
the  stages  employing  the  approach  outlined  above  with  aok=bok=l 

for  k=l,...,7.  In  addition  the  maximum  likelihood  estimation 
procedure  of  Barlow  and  Scheuer  [3]  was  employed  for  purposes  of 
comparison.  Barlow  and  Scheuer  showed  that  the  maximum  likeli- 
hood estimates  of  the  kth  stage  reliability,  , given  ri<=  ...  _<rk 
are  given  by 


= 1 - 


. , k 

Him  £ Xi 

k kli=j  1 


k 

1=3 


They  also  suggested  a method  for  computing  conservative  lower 
confidence  limits  for  reliability  at  each  stage.  This  method 
involved  consideration  of  the  data  as  though  homogeneity 
existed  between  all  stages  as  if  no  reliability  growth  were 
taking  place.  The  usual  technique  for  obtaining  a one-sided 
lower  confidence  limit  for  a binomial  parameter  is  then  used 
with  S k successes  in  Nk  trials  where 


k k 

Sv  = Is-  and  Nv  = I n- 

K i=l  1 k i=l  1 * 


Table  2 provides  the  results  of  application  of  the  two  approaches 
which  include  point  estimates  of  the  reliability  in  each  stage 
for  both  approaches,  lower  Bayesian  90 % confidence  limits, 
and  Barlow  and  Scheuer^  conservative  90$  confidence  bounds. 
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‘ TABLE  2 


STAGE 

BAYESIAN 

ESTIMATE 

M.L.E. 

LOWER  90J&CONF.  ' LIM. 
BAYESIAN  CONSERVATIVE 

1 

0.583 

0.600 

0.401 

0.354 

2 

0.607 

0.550 

0.463 

0.385 

3 

0.728 

0.700 

0.596 

0.467 

4 

0.729 

0.650 

0.613 

0.486 

5 

0.819 

0.800 

0.713 

0.540 

6 

0.891 

0.900 

0.808 

0.594 

7 

0.884 

0.850 

0.808 

0.619 

The  results  Included  In  TABLE  2 reflect  Immediately  the  rel- 
atively  close  agreement  between  the  point  estimates  determined 
using  the  two  approaches.  They  also  demonstrate  the  extreme 
conservatism  of  the  confidence  bound  suggested  by  Barlow  and 
Scheuer  and  the  improvement  in  this  respect  achieved  by  the 
method  introduced  here.  These  features  are  more  readily  apparent 
when  reviewed  graphically,  as  in  FIGURE  1,  where  the  tracking 
of  reliability  growth  for  this  example  is  provided. 

The  improvement  in  the  confidence  bound  obtained  via  the 
Bayesian  approach  can  be  viewed  from  a different  aspect  and 
said  to  result  from  an  increase  in  the  pseudo-sample  size 
for  each  stage.  This  value  for  the  kth  stage  is  denoted  by  Nk 

and  Is  simply  the  sum  of  the  parameters  of  the  beta  approximation 
determined  for  h(r r2£ * . ,<rk  ) reduced  by  the  sum  of  the  prior 

parameters  for  that  stage,  i.e.  aok+t^k  . Hence, 


Nk  ~ Pk  + ~ aok  “ bok  . 


If  we  define  the  number  of  pseudo-successes  to  be  sk=pk-aok  and 
the  number  of  pseudo-failures  to  be  Fk  ~qk  ~b  ok  , then  uk=sk  ~Fk 
The  values  of  ^k , sk  , ^k  for  our  example  are  provided  in  TABLE  3 
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FIGURE 


TABLE  3 


STAGE,  k 

=qk-l 

Nk=Sk  +Fk 

1 

6 . 0 

4.0 

10.0 

2 

10.6 

6.5 

17.1. 

3 

13.2 

4,3 

17.5 

4 

17.3 

5.3 

23.1 

5 

l8.o 

3.2 

21.2 

6 

20.9 

1.7 

22.6 

7 

27.1 

2.7 

29.8 

The  results  In  this  table  show  the  degree  to  which  the  pseudo- 
sample size  was  increased  in  each  stage  as  compared  to  the  actual 
sample  size  of  10  tested  at  each  stage.  This  increase  in  pseudo- 
sample size  and  the  use  of  §k  , rather  than  the  cumulative 
number  of  actual  successes  used  by  Barlow  and  Scheuer  lead  to 
the  improved  confidence  limits  in  TABLE  2. 

In  addition  to  carrying  out  the  necessary  computation 
for  many  examples  such  as  the  one  just  presented,  computer 
simulation  and  direct  computations  were  made  to  determine  the 
properties  of  the  point  estimates  resulting  from  the  suggested 
Bayesian  approach.  These  investigations  involved  the  effects  . 
of  varying  sample  sizes  within  stages  and  from  stage  to  stage. 

A broad  spectrum  of  true  reliability  growth  profiles  were 
employed  for  which  a determination  was  made  of  the  bias  and 
mean-square-error  (MSE)  of  estimates  of  reliability  at  each 
stage.  These  values  were  determined  for  each  of  the  profile/sample 
size  cases  for  both  the  Bayesian  approach  and  the  maximum 
likelihood  method  of  Barlow  and  Scheuer.  In  every  case  at 
each  stage  the  MSE  of  the  estimates  resulting  from  the  Bayesian 
approach  were  smaller  than  that  for  the  MLE.  In  most  cases, 
while  the  bias  of  the  Bayesian  estimates  was  larger  than  that 
of  the  MLE  In  the  early  stages,  as  the  number  of  stages 
increased  the  difference  decreased.  In  some  of  these  cases 
the  bias  of  the  Bayesian  estimates  became  smaller  than  that 
of  the  MLE  in  the  later  stages. 

PROCEDURE  FOR  PREDICTION:  At  any  intermediate  stage  of 

testing  during  a development  program,  the  program  managers 
would  like  to  have  a projection  of  the  reliability  through 
future  stages  of  testing  assuming  the  program  will  proceed 
without  many  unforeseen  problems.  In  this  respect,  a reliability 
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growth  analysis  approach  which  provides  only  a method  of  tracking 
the  past  and  current  status  of  system  reliability,  is  incomplete, 
A means  of  predicting  or  forecasting  reliability  through  later 
stages  of  testing  based  on  the  past  experience  is  necessary. 

Fortunately  there  is  a natural  and  intuitively  pleasing 
way  to  extend  the  Bayesian  approach  of  tracking  reliability 
presented  above  to  provide  a procedure  for  predicting  reliability 
growth  through  future  stages.  Suppose  that  testing  through 
stage  k has  been  accomplished  and  the  resulting  data  has  been 
used  along  with  data  from  previous  stages  to  track  reliability 
through  stage  k by  the  Bayesian  approach.  Prediction  for  future 
stages  is  accomplished  by  continuing  the  analysis  conducted  for 
stages  1 through  k into  stages  k+l,k+2 , . . . ,m,  by  defining  the 
prior  beta  parameters  for  these  stages  to  be 

aoi  ~ Pk  and  boi  = Pk 


for  i=k+l,k+2 , . . . ,m,  where  pk  and  q k are  the  computed  parameters 

of  the  beta  approximation  to  h(rk|^iS£2  = • • *1  *k)  • Then  for  each 

stage  i,  i-k+l,...,m,  since  no  actual  data  has  been  observed, 
we  define  si=o  and  xl  ~o,  so  that  the  parameters  of  the  uncondi- 
tional marginal  beta  distribution  of  the  , i-k+ls...,m  are 


ai  - si  + Pk  = Pk  and  bi  = xi  + 3k  = <3k 


The  conditional  marginal  distribution  of  the  fi  given 

ri,<  r2<;,  . . can  then  be  approximated  by  a beta  distribution 

by  continuing  the  procedure  used  for  stages  1 through  k.  Point 
and  interval  estimates  of  the  predicted  reliability  for  the 
future  stages  can  then  be  obtained  in  the  same  way  as  for  earlier 
stages . 

This  procedure  was  applied  to  the  example  presented  in 
the  previous  section  to  predict  reliability  growth  for  stages 
8,  9,  and  10,  since  for  that  example  k=7.  The  results  are 
presented  in  TABLE  4 and  include  point  estimates  and  lower  90$ 
Bayesian  confidence  limits  for  stage  7,  computed  previously, 
as  well  as,  for  stages  8,  9S  10. 
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RELIABILITY 


STAGE 


TABLE  4 

POINT  LOWER  90$ 

ESTIMATE  CONF.  LIMIT 


7 

8 

9 

10 


0.884 

0.915 

0.936 

0.950 


0.808 

0.864 

0,901 

0.926 


A graphical  representation  of  the  tracking  and  prediction 
of  reliability  growth  for  stages  1 through  10  of  our  example 
is  provided  in  FIGURE  2. 


RELIABILITY  GROWTH-INDEPENDENT  POISSON  PROCESSES: 


We  conclude  this  paper  with  a discussion  of  an  extension 
of  the  method  presented  in  the  previous  section  for  test  stages 
defined  by  independent  Bernoulli  processes  to  the  case  where 
the  test  stages  are  defined  by  independent  Poisson  processes. 

In  such  cases  data,  instead  of  being  provided  as  a number  of 
failures  in  a given  number  of  trials , is  provided  in  terms  of  a 
number  of  failures  occurring  in  some  total  test  time  at  each 
stage.  That  is,  at  stage  k,  N ^ items  are  placed  on  test, 

the  test  being  terminated  when  either  (a)  all  Nk  items  have 
failed,  (b)  a predetermined  number,  Nok , of  items  have  failed, 
or  (c)  a predetermined  test  time,  Tok , has  been  reached.  Re- 
gardless of  the  manner  in  which  the  kth  stage  test  is  terminated, 
the  time  to  failure  of  each  failed  item  is  recorded  along  with 
the  total  number  of  failures,  and  this  information  Is  used  to 
define  a set  of  sufficient  statistics,  { nk,  Tk> , for  the  unknown 

kth  stage  failure  rate,  Ak  . The  definition  of  the  elements 

of  the  sufficient  statistic  set  depend  on  the  manner  in  which 
the  test  was  terminated. 

When  termination  criterion  (a)  is  used  nk  and  Tk  are 
defined  as 


n 


k 


and 


Skt, 

i=l  1 


where  ti  is  the  ith  greatest  time  to  failure  of  the  nk  items. 
For  termination  criterion  (b)  we  have 


nk 

nk  = Nok  and  Tk  = + (Nk  - Nok)tnk 
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with  the  t±  defined  as  for  (a).  For  criterion  (c)  we  have 

nk 

nk  “ Nok  an^  Tk  + ^Nk  - Nok^  Tok 

where  N ^ is  the  observed  number  of  failures  occuring  before 

or  at  T ok  and  the  t±  defined  as  for  (a). 

The  probability  distribution  of  the  times  to  failure  of 
the  items  tested  at  stage  k has  density 

f(t|xk)  = o < t < » 


This  is  the -density  of  the  exponential  distribution  with  failure 
rate  Xk  . It  is  easily  shown  that  the  set  { n^5  T^},  with 

definition  depending  on  the  termination  criterion  is  a set  of 
sufficient  statistics  for  Ak  given  the  sample  information  at 

each  stage.  The  family  of  distribution  conjugate  to  this  Poisson 
process  Is  the  family  of  gamma  distribution  characterized  by 
the  density 


fyU|n,T) 


m)nexp[-TA]  T 
r (n+l) 


0 < A < «> 


Let  stage  k prior  be  f^,(Ak|n^,T£) 

where  { n ' T * } is  Interpreted  as  the  prior  sufficient  statistic 

K.  it 

set,  then  given  the  observed  sufficient  statistic  set  {r^  ,Tk> 
the  posterior  distribution  of  Xk  has  density  f^(Xk|  nk>V’) 


n"  s n.  + n.' 


m r* 


•f  t; 

k 
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t 1 

For  the  remainder  of  this  discussion  we  will  let  nk=Tk=0,  the 
situation  analogous  to  aok *=b ok=  1 for  the  Bernoulli  process.  Then 
the  posterior  distribution  of  Ak  Is  considered  to  have  density 

^ y^kl  n k>  ^k  ^ * 

To  develop  an  approach  to  analysis  of  reliability  growth 
for  this  case  we  first  note  that  the  reliability  of  each  item 
placed  on  test  at  stage  k in  terms  of  a specified  mission  time, 
tm,  is  given  by 

rk  - Pr<  **£  tm}= 


so  that  if  we  treat  the  unknown  failure  rate  as  a Bayesian  random 
variable,  Ak  , then  the  reliability  is  also  such  a random  variable 

defined  by 


= e'^kV 


with  distribution  depending  on  the  distribution  of  xk.  Instead 

of  determining  the  distribution  of  the  fk  directly  and  proceeding 

with  our  analysis  of  reliability  growth  with  the  assumption 
that  <^2^  ♦ • . we  wil1  approximate  these  distributions  by 

members  of  the  beta  family  using  the  method  of  moments.  This 
is  accomplished  by  first  determining  the  first  and  second  moments 
of  £k  , denoted  ylk  and  y2k  , by 


Mlk  " E(rk)  = E(e'JkS  = /“  (Xk|nk,Tk)  dAk 


and 


= E(r£)  = E(e"2^ktm)  = /“  e"2Xktmf  (A,  In,  ,T.  ) dA 


'2k  k 


y k 1 k ' k k 
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Performing  the  required  Integration  yields 

Ulk  = ' V(tm  + Tk)lnk+1  " I V(1  + Vlnk+1 

and 

**2k  = f V<2tm  + Tk>lnk+1  = f \/(2  + 6k)lnk+1 

where  is  the  ratio  of  total  test  time  to  mission  time. 

We  can  then  employ  (12)  and  (13)  with  ak  and  bk  substituted 
for  pk  and  qk  , respectively,  to  obtain  the  parameters  of  the 

beta  fit  to  the  distribution  of  rk  . The  resulting  beta 

distributions  for  each  stage,  f B (r k| % ,bk ) , can  then  be  used  for 

reliability  growth  analysis  proceeding  in  exactly  the  same  way 
as  for  the  Bernoulli  process  to  obtain  point  and  interval 
estimates  of  the  reliability  at  each  stage  under  the  non- 
decreasing reliability  assumption.  It  is  expected  that  simu- 
lation studies  would  demonstrate  that  these  estimates  have 
properties  almost  as  good  as  those  demonstrated  for  the 
Bernoulli  process. 
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EXPERIMENTAL  COLLECTION  OF  STATISTICS  BY- 
COMPUTER  SIMULATION!  THE  AUTOVON  NETWORK1 
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ABSTRACT,  A computer  simulation  model  of  the  AUTOVON  Network  has 
been  constructed  in  GPSS-V,  It  Includes  all  70  switches  and  60  multiple- 
homed  PBX*s  in  the  continental  United  States,  This  model  Is  used  to 
predict  the  behavior  of  the  network  under  normal  operating  conditions 
and  under  stress  situations,  such  as  a sudden  accumulation  of  high- 
priority  calls  or  the  elimination  of  a number  of  switches  or  intercon- 
necting links.  The  size  of  the  network  and  its  Importance  within  the 
communications  systems  make  It  impractical  to  determine  the  reactions 
of  the  network  by  actual  experiment.  The  large  degree  of  complication 
brought  about  by  the  multiple  Interconnections,  routing  instructions 
and  five  levels  of  call  priority  preclude  a purely  theoretical  calcula- 
tion, Thus,  the  tool  of  computer  simulation  provides  the  best  way  to 
obtain  the  answers  when  the  network  is  represented  in  the  required 
detail.  Questions  that  are  answered  in  this  manner  include  the  effects 
of  routine  traffic  on  the  high-priority  calls  under  stress  condition, 
and  the  transient  effects  of  the  elimination  of  a block  of  switches  or 
inter switch  links.  The  validity  of  the  statistics  gathered  for  the 
model  depends  on  the  length  of  the  simulation  run  and  the  adequacy  of 
the  random  number  generators  used.  General  results  on  these  points  are 
not  known  for  GPSS, 

1,  INTRODUCTION,  The  AUTOVON  network  is  part  of  the  World-Wide 
Military  Command  and  Communications  System/Defense  Communications 
System  (WWMCCS/DCS) , The  Continental  US  (CONUS)  part  of  the  network 
covers  the  US  and  Canada  and  permits  voice  communications  for  routine 
traffic  and  priority  matters. 

The  precise  configuration  of  the  network  is  constantly  being 
altered  as  a result  of  improvements  and  changes  in  the  nature  of  the 
load.  The  model  that  was  used  in  the  simulation  consists  of  70  switches 
and  60  multiple-homed  Private  Branch  Exchanges  (PBX) . For  this  investi- 
gation, the  changes  introduced  in  the  network  do  not  affect  the  validity 
of  the  results. 

The  model  is  coded  in  GPSS-V,  and  it  faithfully  represents  the 
processing  of  long-distance  calls  through  the  AUTOVON  system.  Minor 
changes  are  introduced  to  simulate  the  special  conditions  considered 
for  each  run. 

The  purpose  of  these  runs  was  to  study  transient  effects  on  the 
flow  of  traffic,  which  may  occur  under  emergency  conditions.  This  is 


1 


This  effort  was  initially  formulated  as  part  of  DNA/DCA  joint  PREMPT  Program 


323 


in  contradistinction  to  the  more  usual  simulation  programs  primarily 
directed  to  the  evaluation  of  steady  state  conditions  of  normal  op- 
erations. 


Section  2 presents  a summary  of  the  functional  aspects  of  the 
network  and  section  3 describes  the  principal  features  of  the  model. 

Section  4 summaries  a study  of  the  effects  of  routine  background  traffic 
on  surges  of  high-priority  calls;  and  section  5 gives  some  results  for 
the  transient  effects  of  the  elimination  of  a block  of  switches  or  a part 
of  the  link  between  them  on  priority  traffic.  Some  of  the  problems 
associated  with  the  statistical  aspects  of  simulations  are  discussed  in 
section  6, 

The  effect  of  the  routine  background  traffic  is  not  large  but 
clearly  noticeable,  about  10%,  The  transient  effects  of  the  disruption 
of  switches  or  links  are  negligible,  and  the  steady-state  effects  depend 
strongly  on  the  selection  of  links  to  be  eliminated.  This  reflects  the 
importance  of  the  role  of  the  random-number  generators  in  a GPSS  model, 

2,  THE  AUTOVON  NETWORK,  There  are  70  switches  that  form  the 
basic  grid  of  AUTOVON,  ideally  laid  out  in  a hexagonal  pattern.  Each 
switch  then  has  6 nearest  neighbors  and  6 second  nearest  neighbors, 
except  for  those  on  the  edges  of  the  network,  and  each  switch  is  con- 
nected to  these  12  neighbors  that  form  the  home  grid  of  the  switch. 

In  practice  this  pattern  is  considerably  distorted  and  the  home  grid 
can  include  either  more  or  less  than  12  switches.  In  addition,  each  switch 
is  connected  to  a number  of  other  switches  outside  the  home  grid  by 
long  lines.  The  60  multiple-homed  PBX's  are  smaller  switches  that  are 
connected  to  two  (in  one  case,  three)  main  switches  and  have  a single  address. 
When  the  destination  of  a call  is  established,  each  switch  has  a table 
that  shows  how  to  route  this  call  to  a switch  or  PBX  in  the  most  expe- 
ditious manner.  Outside  the  home  grid,  the  routing  instructions  provide 
a direct  route,  when  available,  and  three  triples  of  switches  that  serve 
as  an  intermediate  destination,  a most  direct  triple,  a first  alternate 
triple,  and  a second  alternate  triple.  Inside  the  home  grid,  there 
exist  a direct  route  and  a first  and  second  alternate  triples  only.  The 
home  grid  for  a multiple-homed  PBX  consists  of  the  home  grids  of  the 
switches  to  which  it  is  connected. 

There  are  five  priority  levels  for  the  calls,  routine  calls  having 
the  lowest  priority.  The  routing  instructions  for  routine  calls  restrict 
them  in  their  ability  to  move  laterally  when  forward  routes  are  not 
available;  for  priority  calls,  a route-control  digit  is  used  to  avoid 
shuttling  and  circular  patterns  in  lateral  moves.  Calls  can  preempt 
trunks  used  by  those  of  lower  priorities  after  searching  for  an  avail- 
able one  to  the  destination  or  in  the  most  direct  triple.  The  PBXfs  are 
normally  capable  of  generating  only  routine  traffic,  and  they  can  go  to 
either  switch  first,  whereas  priority  calls  typically  originate  from 
subscribers  directly  connected  to  a switch. 


The  information  to  process  a call  is  transmitted  forward  from  one 
switch  to  the  next  as  it  uses  tandem  switches  to  reach  its  destination. 
Although  part  of  this  information  is  the  priority,  priority  calls  get 
no  preferential  treatment  when  queuing  for  common  equipment  used  in 
call  processing  at  a switch, 

A more  detailed  description  of  the  AUTOVON  network  and  its 
simulation  will  be  presented  in  an  HDL  Technical  Report  now  in  preparation, 

3,  THE  NETWORK  MODEL,  The  code  for  simulation  of  the  AUTOVON 
network  is  written  in  GPSS-V.  It  uses  the  random  number  generators  in 
the  origination  of  calls,  the  selection  of  destination,  the  delays  at 
switches,  the  length  of  conversation,  the  selection  of  a tandem  switch 
within  a triple  for  priority  calls,  the  choice  of  a call  to  be  preempted, 
etc*,  modified  in  the  appropriate  way  for  each  case. 

The  background  traffic  is  generated  in  such  a way  that  the  network 
is  more  or  less  uniformly  loaded,  as  indicated  by  the  number  of  calls 
blocked  at  each  switch.  To  get  good  statistics  for  reasonable  run 
times,  the  load  is  chosen  rather  heavy,  so  that  an  appreciable  fraction 
of  the  calls  is  blocked  or  preempted;  this  is  not  intended  to  represent 
the  normal  traffic  pattern.  The  priority  profile  of  the  calls  is  also 
chosen  in  a way  that  it  facilitates  the  study  of  the  special  effects 
that  are  being  Investigated,  and  does  not  represent  an  average  condition. 

The  model  uses  the  actual  routing  instructions  for  the  network, 
and  the  numbers  of  interswitch  and  PBX-to-switch  trunks  are  close  to  the 
actual  ones.  It  also  follows  the  procedures  used  by  the  switch  to  preempt 
calls* 


A switch  is  represented  by  a delay,  which  in  this  case  is  chosen 
to  be  a function  of  the  load.  This  function  was  determined  by  a simu- 
lation of  an  AECo  switch  in  a separate  model;  it  was  assumed  that  the 
differences  between  this  and  the  ESS  and  4W5  switches,  which  are  the 
two  other  kinds  in  the  network,  would  not  affect  the  nature  of  the  re- 
sults* It  is  straightforward  to  include  a more  detailed  model  of  the 
switch  in  the  simulation  if  this  is  necessary*  The  local  traffic  that 
might  exist  and  the  possibility  of  the  called  line  being  busy  were  not 
taken  into  account.  This  could  be  included  by  a suitable  extension  of 
the  code.  Additional  assumptions  are  detailed  in  the  following  sections, 

4,  EFFECTS  OF  BACKGROUND  LOAD  ON  PRIORITY  TRAFFIC,  Although  the 
ability  of  high-priority  calls  to  preempt  lower  priority  ones  seems  to 
indicate  that  routine  traffic  has  no  effect  on  the  priority  traffic, 
there  are  at  least  two  reasons  why  a high  load  of  background  traffic 
makes  the  completion  of  priority  calls  more  difficult. 

The  priority  of  a call  is  recognized  only  by  a switch  after  this 
information  has  been  relayed  to  it  by  the  previous  switch.  Thus,  while 
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queuing  up  to  be  serviced  for  the  first  time  by  the  switch  marker, 
routine  calls  have  the  same  probability  of  being  taken  as  a priority 
one,  delaying  their  processing.  Even  hot-line  calls,  which  are  checked 
first  by  the  marker  in  the  originating  switch,  receive  no  special  pref- 
erence when  competing  with  other  calls  to  be  recognized  at  another  switch. 
Furthermore,  there  is  a time-out  limit  for  the  call  to  be  accepted  by 
the  next  switch.  If  this  time  is  exceeded,  the  previous  switch  will  try 
another  trunk;  if  this  happens  again,  it  decides  that  the  switch  is  out 
and  tries  another  route. 

Routine  traffic  also  affects  the  routing  of  priority  calls  through 
the  idle  search.  If  a direct  line  is  available  but  all  trunks  are  busy, 
a priority  call  will  normally  search  for  a free  trunk  in  the  most  direct 
triple  of  switches;  only  if  none  is  available  does  it  go  back  to  preempt 
a lower  priority  call  in  the  direct  route.  The  time  delay  for  the  search 
itself  is  negligible;  but  when  a route  from  the  triple  is  chosen  the  number 
of  tandem  switches  increases,  the  call  has  to  be  processed  by  the  additional 
switches,  more  equipment  is  used,  and  the  probability  of  its  being  preempted 
by  a call  of  higher  priority  also  increases. 

To  determine  the  extent  of  these  effects,  the  system  was  heavily 
loaded  with  routine  traffic.  At  a set  time,  it  is  assumed  that  an 
emergency  occurs  and  that  a very  large  number  of  priority  calls  is  generated. 
At  the  same  time,  most  of  the  routine  traffic  is  excluded  from  AUTOVON; 
in  the  model,  the  remaining  routine  traffic  being  generated  is  repre- 
sented by  the  50%  of  blocked  and  preempted  calls  that  try  again.  It 
is  also  assumed  that  an  additional  40%  of  the  priority  traffic  originates 
in  the  Washington,  DC  area,  and  that  an  additional  30%  terminates  there.  No 
priority  traffic  originates  from  the  multiple-homed  PBXfs,  but  40%  termi- 
nates there.  Statistics  are  taken  primarily  for  the  priority  calls. 

The  surge  of  calls  is  of  short  duration,  and  the  network  is  then  allowed 
to  go  to  an  idle  condition.  After  resetting  the  random  number  generators, 
the  same  priority  calls  are  generated,  this  time  on  an  empty  network, 
and  the  statistics  are  collected  separately  and  compared  with  the  previous 
ones. 


Before  giving  some  of  the  results,  a few  additional  details  on  the 
simplifying  assumptions  for  the  switch  behavior  are  in  order.  The  delay 
of  the  call  at  a switch  is  a function  of  the  load,  which  was  computed 
for  the  standard  AECo  switch  with  24  registers,  with  the  proper  differences 
for  the  originator  and  destination  switches.  This  value  is  then  increased  or 
decreased  at  random  by  up  to  20%  to  give  some  expression  of  the  arbitrary 
selection  of  a call  by  the  marker.  The  switch  can  handle  only  24  calls  at  a 
time,  and  new  calls  have  to  queue  for  the  marker.  It  was  assumed  that  a queue 
of  5 calls  or  less  would  make  it  try  one  trunk,  of  10  calls  or  less,  two 
trunks;  and  for  more  than  10  calls.  It  continues  its  search  for  another 
route.  A limit  of  56  calls  queued  was  set  for  the  originator  switch, 
representing  the  patience  of  the  caller  to  get  a dial  tone.  These  as- 
sumptions do  not  reflect  precisely  the  variations  in  delay  times,  but  a 
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better  solution  would  have  demanded  a rudimentary  switch  model  to  be 
included,  with  the  corresponding  increase  in  running  time.  All  these 
additional  waiting  periods  increase  the  delay  time  at  the  switch* 

In  each  case,  5430  priority  calls  are  generated  with  a uniform 
priority  profile,  and  the  number  of  calls  started  increases  to  7146  and 
7141  (with  and  without  background  of  routine  calls)  due  to  retries  from 
blocked  and  preempted  calls*  The  calls  that  reached  their  destination 
were  2227  and  2366  respectively,  an  increase  of  approximately  6%  when 
there  was  no  interference  from  routine  traffic*  The  numbers  of  completed 
calls  were  3573  and  2085  respectively,  but  the  first  number  includes 
routine  calls;  in  the  second  case,  38%  of  the  calls  were  eventually 
completed.  It  should  be  noted  that  this  percentage  does  not  correspond 
to  a realistic  scenario,  but  to  an  overloaded  network.  The  average 
number  of  tandem  switches  used  by  priority  calls  that  were  completed  is 
1.561  and  1.474  respectively,  showing  a decrease  of  6%;  the  maximum  number 
of  tandem  switches  was  6 in  both  cases.  The  average  time  for  a priority 
call  to  reach  its  destination  was  120  and  111  seconds,  showing  a decrease 
of  8%,  An  indication  of  the  overloading  of  the  network  is  the  average 
number  of  calls  active  or  queued  for  a switch  when  a new  call  arrives, 
which  is  25.3  and  24,7  calls  respectively,  with  a maximum  of  81  as  should 
be  expected  from  the  patience  assigned  to  the  caller. 

The  number  of  blocked  priority  calls  shows  a decrease  of  10%,  from 
2930  to  2650,  while  the  corresponding  number  of  preempted  calls  shows 
an  increase  of  9%  from  856  to  928,  This  last  result,  which  at  first 
sight  seems  to  go  against  the  general  trend  of  improvement  with  the 
absence  of  background  traffic,  probably  results  from  the  lack  of 
preemptable  routine  calls  and  the  increase  in  the  number  of  established 
priority  calls.  The  largest  effect  appears  in  the  number  of  highest 
priority  calls  that  were  blocked,  with  an  11%  decrease  from  681  to  613. 

For  the  next  lower  priority,  the  decrease  is  a 7%  from  753  to  697,  ac- 
companied by  a 10%  increase  in  preempted  calls,  from  137  to  151, 

The  conclusion  that  can  be  drawn  from  the  above  results  is  that 
the  effects  of  routine  background  traffic  are  small  but  not  negligible, 

5.  SWITCH  OR  LINK  ELIMINATION.  To  test  the  transient  effects  of 
the  elimination  of  a block  of  12  switches  in  the  center  of  the  network, 
three  sets  of  statistics  are  taken.  The  first  set  is  for  normal  con- 
ditions, the  second  one  is  right  after  the  switches  have  been  eliminated, 
and  the  third  one  is  taken  after  the  network  has  reached  a steady  state 
without  the  switches.  The  network  is  heavily  loaded  with  routine  traffic 
and  priority  traffic  of  the  three  lower  priorities,  and  superimposed  is 
eoast-to-coast  traffic  of  the  highest  priority  used  to  test  more  vividly 
the  effects  of  the  switch  elimination.  The  statistics  are  gathered  for 
a simulated  time  of  6 minutes  each  time. 

The  numbers  of  special  calls  (highest  priority,  coast  to  coast)  for 
the  three  intervals  are  580,  571,  and  635  respectively,  and  the  numbers 
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of  calls  that  were  blocked  during  these  periods  were  23,  177  and  172, 

It  is  difficult  in  this  case  to  establish  a clear  relationship  between 
the  two  sets  of  numbers,  since  the  former  set  does  not  include  calls  that  try 
again  and  there  is  a variable  time  delay  between  the  initiation  of  the 
call  and  its  reaching  the  destination  switch  or  PBX,  The  average  time  to 
connect  the  special  calls  was  13,1,  13,3,  and  13,3  seconds;  the  average  numbers 
of  tandem  switches  used  by  these  calls  were  1,425,  1,415,  and  1,441,  with 
maximum  numbers  of  4,  5,  and  5 respectively. 

There  is  a considerable  increase  in  the  number  of  blocked  calls 
(again  it  should  be  emphasized  that  the  load  is  not  realistic  but  chosen 
large  enough  to  bring  out  these  effects),  but  there  is  no  sizable  effect 
on  those  calls  that  get  through  either  in  the  period  immediately  following 
the  perturbation  or  later  on.  No  appreciable  queuing  occurs  at  the 
switches,  where  the  maximum  number  of  active  calls  being  processed  goes 
from  12  to  15  to  IB  in  the  three  periods,  compared  with  a maximum  capacity 
for  24  simultaneous  calls  of  the  standard  AECo  switch. 

The  number  of  "preempted”  calls,  all  of  lower  priority,  goes  from 
720  to  1272  to  570;  the  second  number  includes  those  calls  that  were  dis- 
connected due  to  the  failures, 

A variation  of  the  same  problem  was  then  tried,  where  the  links 
between  the  same  12  switches  and  other  switches  were  failed  with  a prob- 
ability of  ,3,  as  chosen  by  a random  number  generator. 

The  special  calls  were  580,  537,  and  561;  those  blocked  were  23,  62, 
and  108;  the  average  times  to  connection  were  13.1,  13.3,  and  13.3  seconds; 
the  average  number  of  tandem  switches  1,425,  1,467,  and  1,494,  with  a maximum 
of  4,  5,  and  6 switches  respectively. 

There  is  a significant  reduction  in  the  number  of  blocked  calls  when 
the  switches  remain  in  service,  but  otherwise  the  behavior  of  the  network 
is  not  too  different  from  the  former  case  or  the  normal  network.  The  large 
change  in  the  number  of  blocked  calls  from  the  transient  state  to  the  steady 
state  of  the  damaged  network  is  due  to  a change  in  the  sequence  of  random 
numbers  to  determine  the  elimination  of  links.  In  the  run  to  determine  the 
transient  effects,  71  links  were  eliminated,  while  in  the  one  for  steady-state 
conditions  72  were  eliminated,  but  they  were  not  the  same  ones. 

The  numbers  of  "preempted"  calls  were  720,  1014,  and  629  respectively, 
and  the  maximum  active  loads  on  the  switches  were  12,  13,  and  12  calls. 

6,  STATISTICAL  CONSIDERATIONS,  In  conducting  computer  "experiments" 
such  as  these,  it  is  important  to  keep  in  mind  the  requirements  and 
limitations  of  the  simulation  language  used. 

There  always  exists  a problem  about  the  length  of  the  simulation 
run  required  to  gather  the  information  sought.  It  is  desirable 
to  ascertain  that  the  results  are  reasonably  stable,  which  would  suggest 
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repeating  the  runs  with  different  time  spans  and  random  number  generators. 

But  on  a big  model  such  as  that  on  the  AUTOVON  network,  which  uses  between 
600  and  800K  of  core  and  about  one-half  the  simulated  time  in  CPU  time 
on  an  IBM  360-95,  there  are  strong  reasons  to  limit  the  length  of  runs 
to  a minimum*  (The  CPU  time  can  vary  significantly  with  the  simulated  load  of 
the  system,  the  type  of "experiment" , and  even  the  fortune  of  running  in 
a faster  portion  of  core.)  Experience  then  dictates  the  length  of 
simulation  runs,  together  with  the  characteristic  times  in  the  model 
such  as  the  delays  at  the  switches,  the  length  of  conversations,  and  the 
intercall  arrival  time. 

Questions  also  arise  about  the  influence  of  a particular  sequence 
of  numbers  from  a random  number  generator,  which  can  be  considerable 
as  shown  by  the  results  from  the  elimination  of  links  discussed  In 
section  5.  This  problem  can  be  associated  with  the  limitations  of  a 
random  number  generator  or  with  the  actual  uncertainties  of  the  system 
itself  when  chance  plays  an  important  role. 

Another  problem  arises  when  statistics  gathered  under  different 
conditions  are  compared.  It  appears  wasteful  to  extend  the  length  of 
the  runs  until  the  averages  are  well  established,  but  the  manipulation 
of  the  random  number  generators  to  reproduce  the  same  sequences  requires 
a considerable  amount  of  programming  with  the  associated  drawbacks. 

At  the  present  level  of  understanding  of  these  characteristics  of 
a simulation  language,  these  decisions  have  to  be  left  to  the  judgement 
of  the  developer  of  the  model  both  in  terms  of  the  program  characteris- 
tics and  the  interpretation  of  the  results, 

7,  SUMMARY  AND  CONCLUSIONS.  A detailed  GPSS-V  model  of  the  CONUS 
AUTOVON  network  was  developed,  and  it  was  used  to  determine  some  load 
effects  and  transient  effects  under  abnormal  conditions. 

It  was  found  that  routine  background  traffic  has  a small  but  signif- 
icant obstructing  role  on  a surge  of  priority  traffic.  The  most  critical 
phase  of  processing  the  call  occurs  after  a trunk  to  another  switch  is 
taken  and  the  call  waits  for  a response  from  this  switch.  When  it  is 
overloaded,  a call  might  not  be  recognized  soon  enough  to  avoid  a time-out 
which  delays  the  calls  or  results  in  its  being  blocked  when  trunks  are 
really  still  available.  Other  calls  are  significantly  delayed  or 
abandoned  because  they  get  no  dial  tone.  Another  cause  of  delays  is 
the  idle  search  for  trunks  in  the  most  direct  triple  when  calls  of  lower 
priority  are  using  the  direct  trunks.  The  deterioration  of  the  service 
to  priority  calls  due  to  the  presence  of  background  traffic  is  of  the 
order  of  10%. 

The  elimination  of  a block  of  12  switches  in  the  center  of  the 
network  reduces  markedly  the  probability  of  a coast-to-coast  high 
priority  call  being  completed  during  heavy  traffic,  but  it  does  not 
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increase  by  a large  amount  the  time  it  needs  for  processing  or  the  number  of 
tandem  switches  it  uses.  The  transient  effects  due  to  the  traffic  that  is 
disrupted  and  tries  again  is  not  a very  significant  factor  in  the  pro- 
cessing of  the  high  priority  calls.  Similar  results  are  true  when 
about  30%  of  the  links  between  the  switches  are  eliminated  without  af- 
fecting the  performance  of  the  switches  themselves;  in  this  case,  an 
Important  factor  is  apparently  which  of  the  links  are  eliminated. 

These  are  just  a few  special  conditions  that  can  be  simulated  with 
the  computer  model,  where  practical  considerations  preclude  actual 
testing  of  the  network  under  abnormal  circumstances.  Furthermore,  the 
model  can  be  refined  to  include  local  traffic  and  a switch  model,  either 
the  same  one  for  all  switches  or  a different  one  in  each  case. 
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AN  ANALYSIS  OF  BUFFERS  IN  A PRODUCTION  SYSTEM 
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ABSTRACT ; BUFFERS  CAN  BE  USED  TO  INCREASE  THE  AVERAGE  THROUGHPUT 
OF  A PRODUCTION  SYSTEM,  THE  USE  OF  BUFFERS  HAS  THE  EFFECT  OF  SMOOTH- 
ING THE  PROCESS  FLOW  THROUGH  THE  SYSTEM  BY  PROVIDING  FOR  IN-PROCESS 
INVENTORIES.  IF  A BUFFER  EMPTIES,  IT  CAUSES  THE  SHUT-DOWN  OF  OTHER 
MACHINES  IN  THE  PRODUCTION  LINE.  THESE  SHUT-DOWNS  CAN  BE  MINIMIZED 
WITH  PROPERLY  SIZED  BUFFERS. 

INITIALLY  THE  USE  OF  BUFFERS  WAS  INVESTIGATED  TO  DETERMINE  THE 
EFFECTS  THEY  WILL  HAVE  ON  A PRODUCTION  SYSTEM.  RULES  WERE  ESTABLISHED 
TO  IMPROVE  THE  EFFECTIVENESS  AND  INCREASE  THE  BENEFIT  DERIVED  FROM 
PLACING  BUFFERS  IN  A PRODUCTION  LINE.  THESE  RULES  TAKE  INTO  ACCOUNT 
THE  DISTRIBUTION  OF  THE  MTTR  AND  THE  PRODUCTION  SPEED  FOR  EACH  MACHINE 
IN  THE  PRODUCTION  LINE.  THESE  RULES  WERE  THEN  APPLIED  TO  THE  SCAMP 
PRODUCTION  SYSTEM.  A MONTE  CARLO  SIMULATION  OF  THE  SCAMP  SYSTEM  WAS 
USED  TO  DETERMINE  WHAT  EFFECT  VARIOUS  SIZE  BUFFERS  AND  SMALL  CHANGES 
IN  THE  PRODUCTION  SPEED  WILL  HAVE  ON  THE  THROUGHPUT  OF  THE  SYSTEM, 

AND  THUS  MINIMIZING  THE  NUMBER  OF  SHUT-DOWNS  CAUSED  BY  THE  EMPTYING 
OF  A BUFFER. 

1.  INTRODUCTION:  BUFFERS  CAN  BE  USED  TO  INCREASE  THE  AVERAGE 

THROUGHPUT  OF  A PRODUCTION  SYSTEM.  THE  USE  OF  BUFFERS  HAS  THE  EFFECT 
OF  SMOOTHING  THE  PRODUCT  FLOW  THROUGH  THE  SYSTEM  BY  PROVIDING  FOR  IN- 
PROCESS  INVENTORIES.  IF  A BUFFER  EMPTIES,  IT  CAUSES  THE  SHUTDOWN  OF 
OTHER  MACHINES  IN  THE  PRODUCTION  SYSTEM.  THESE  SHUTDOWNS  CAN  BE  MINI- 
MIZED WITH  PROPERLY  SIZED  BUFFERS. 

INITIALLY,  THE  USE  OF  BUFFERS  WAS  INVESTIGATED  TO  DETERMINE  THE 
EFFECTS  THEY  WILL  HAVE  ON  A PRODUCTION  SYSTEM.  RULES  WERE  ESTABLISHED 
TO  IMPROVE  THE  EFFECTIVENESS  AND  INCREASE  THE  BENEFIT  DERIVED  FROM 
PLACING  BUFFERS  IN  A PRODUCTION  LINE.  THESE  RULES  TAKE  INTO  ACCOUNT 
THE  DISTRIBUTION  OF  THE  MEANTIME  TO  REPAIR  (MTTR)  AND  THE  PRODUCTION 
SPEED  FOR  EACH  MACHINE  IN  THE  PRODUCTION  SYSTEM,  AND  STATE  GENERAL 
CONCLUSIONS  AND  RECOMMENDATIONS  CONCERNING  THE  SIZE  OF  THE  BUFFERS 
WHICH  SHOULD  BE  EMPLOYED.  THESE  RULES  WERE  THEN  APPLIED  TO  THE  SMALL 
CALIBER  AMMUNITION  MODERNIZATION  PROGRAM  (SCAMP)  PRODUCTION  SYSTEM. 

A MONTE  CARLO  SIMULATION  OF  THE  SCAMP  SYSTEM  WAS  USED  TO  DETERMINE 
WHAT  EFFECT  VARIOUS  SIZE  BUFFERS  AND  SMALL  CHANGES  IN  THE  PRODUCTION 
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SPEED  WILL  HAVE  ON  THE  AVERAGE  THROUGHPUT  OF  THE  SYSTEM,  AND  THUS 
REDUCE  THE  NUMBER  OF  SHUTDOWNS  CAUSED  BY  THE  EMPTYING  OF  A BUFFER. 

THE  SCAMP  PRODUCTION  SYSTEM  IS  DESIGNED  TO  PRODUCE  SMALL  CALIBER 
AMMUNITION  AT  A MAXIMUM  RATE  OF  1200  PIECES  PER  MINUTE.  THE  SYSTEM 
IS  A SERIES  OF  MACHINES  KNOWN  AS  SUBMODULES  THAT  PERFORM  UNIQUE 
PROCESSES  THAT  ULTIMATELY  YIELD  A COMPLETED  CARTRIDGE.  THE  MACHINES 
CONSIST  OF  A CASE  SUBMODULE  THAT  PRODUCES  A COMPLETE  BRASS  CARTRIDGE 
CASE;  A PRIMER  INSERT  SUBMODULE  THAT  INSERTS  THE  PRIMER  INTO  THE  JUST 
MANUFACTURED  CASE;  A BULLET  SUBMODULE  THAT  PRODUCES  THE  COMPLETED 
BULLET;  AND  A LOAD  AND  ASSEMBLE  SUBMODULE  THAT  LOADS  THE  CASE  WITH 
PROPELLANT  AND  INSERTS  THE  BULLET  INTO  THE  CASE,  THUS  PRODUCING  A 
COMPLETED  CARTRIDGE. 

BECAUSE  OF  THE  SERIAL  NATURE  OF  THIS  PRODUCTION  SYSTEM,  IT  WAS 
FELT  THAT  SOME  IN-PROCESS  INVENTORIES  BETWEEN  SUBMODULES  WOULD  BE  RE- 
QUIRED IN  ORDER  TO  OBTAIN  A SYSTEM  THROUGHPUT  THAT  REFLECTED  THE  CAPA- 
BILITIES OF  THE  EQUIPMENT.  THE  ADDITION  OF  BUFFERS  REDUCES  THE  SERIAL 
EFFECT  AND  SMOOTHS  THE  PRODUCT  FLOW  THROUGH  THE  SYSTEM.  WITH  BUFFERS, 

A SUBMODULE  FAILURE  WILL  NOT  AUTOMATICALLY  SHUTDOWN  THE  WHOLE  SYSTEM, 

BUT  WILL  ALLOW  CONTINUED  PRODUCTION  BY  THE  OTHER  SUBMODULES. 

THE  USE  OF  ARBITRARILY  LARGE  OR  INFINITE  BUFFERS  WILL  PRODUCE  THE 
MAXIMUM  SYSTEM  THROUGHPUT,  WHICH  WILL  BE  THE  THROUGHPUT  OF  THE  LAST 
SUBMODULE  IN  THE  SYSTEM,  FAILURES  IN  ANY  OF  THE  SUBMODULES  WOULD 
NOT  AFFECT  SYSTEM  THROUGHPUT  OR  CAUSE  SHUTDOWN  OF  ANY  OTHER  SUBMODULE 
BECAUSE  THE  INPUT  BUFFER  TO  THE  FAILED  SUBMODULE  CAN  STORE  AN  ARBI- 
TRARILY LARGE  NUMBER  OF  ITEMS  FROM  THE  PRECEDING  SUBMODULE,  THUS  THE 
PRECEDING  SUBMODULE  WILL  NEVER  NEED  TO  BE  SHUTDOWN  DUE  TO  A FULL  BUFFER, 
SIMILARLY,  THE  SUCCEEDING  SUBMODULE  WILL  NOT  BE  SHUTDOWN  SINCE  AN  IN- 
FINITE AMOUNT  OF  IN-PROCESS  WORK  PIECES  ARE  AVAILABLE  AS  INPUT  TO  THAT 
SUBMODULE. 

OBVIOUSLY,  INFINITELY  LARGE  BUFFERS  ARE  IMPRACTICAL.  THIS  PAPER 
ATTEMPTS  TO  ESTABLISH  HOW  BUFFERS  CAN  BE  SIZED  IN  ORDER  TO  MINIMIZE 
THE  SERIAL  EFFECT  OF  A PRODUCTION  SYSTEM  AND  THUS  INCREASE  THE  AVERAGE 
THROUGHPUT  FOR  THAT  SYSTEM. 

2.  METHODOLOGY: 

INITIALLY,  A STATISTICAL  APPROACH  WAS  ATTEMPTED  IN  ORDER  TO  OBTAIN 
NUMERICAL  PROBABILITIES  AND  ESTABLISH  CONFIDENCE  INTERVALS  FOR  THE 
EMPTYING  OF  A BUFFER  FOR  A GIVEN  SIZE.  THIS  METHOD  PROVED  VERY  CUMBER- 
SOME, COMPLEX,  AND  MATHEMATICALLY  TEDIOUS.  AN  ATTEMPT  AT  AN  EXACT 
SOLUTION  ALSO  ELIMINATED  THE  POSSIBILITY  OF  INVESTIGATING  VARIOUS 
ALTERNATIVES,  THEREFORE,  THIS  APPROACH  WAS  ABANDONED. 

A SECOND  METHOD  OF  ATTACHING  A COMPLEX  PROBLEM  IS  TO  MAKE  AN  AB- 
SOLUTE MINIMUM  NUMBER  OF  ASSUMPTIONS  INVOLVING  OPERATING  POLICIES  AND 
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TIME  DISTRIBUTIONS  OF  TIME  TO  FAIL  AND  REPAIR.  THIS  APPROACH  MORE 
ACCURATELY  REPRESENTS  THE  SYSTEM  TO  BE  MODELED  AND  PROVIDES  A GREAT 
AMOUNT  OF  FLEXIBILITY  TO  VARIOUS  TYPES  OF  ASSUMPTIONS  THAT  NEED  TO 
BE  INVESTIGATED.  THE  DISADVANTAGE  OF  THIS  APPROACH  IS  THAT  THE 
GENERALITY  AND  FLEXIBILITY  INCORPORATED  INTO  THE  ANALYSIS  PREVENT 
A COMPLETE  AND  GENERAL  SOLUTION  FOR  THE  DETERMINATION  OF  THE  OPTIMUM 
BUFFER  SIZE. 

THE  ACTUAL  METHOD  OF  ANALYSIS  USED  WAS  A COMBINATION  OF  THE  TWO 
ABOVE  MENTIONED  TECHNIQUES.  INITIALLY,  A GENERALIZED  ANALYSIS  OF 
THE  EFFECT  OF  BUFFERS  IN  THE  SCAMP  PRODUCTION  SYSTEM  WAS  CONDUCTED. 

THIS  RESULTED  IN  THE  ESTABLISHMENT  OF  A SET  OF  RULES  TO  IMPROVE  THE 
EFFECTIVENESS  AND  INCREASE  THE  BENEFIT  DERIVED  FROM  PLACING  BUFFERS 
IN  A PRODUCTION  LINE.  THESE  RULES  TAKE  INTO  ACCOUNT  THE  DISTRIBU- 
TION OF  TIME  TO  REPAIR,  PRODUCTION  SPEEDS,  AND  AVAILABILITY  FOR  EACH 
SUBMODULE  IN  THE  SYSTEM,  AND  WERE  USED  AS  GUIDELINES  AND  HELPED  TO 
LIMIT  THE  NUMBER  OF  ALTERNATIVE  SOLUTIONS. 

IN  ORDER  TO  DETERMINE  NUMERICAL  SOLUTIONS  FOR  THE  BUFFER  SIZES, 

A MONTE  CARLO  SIMULATION  OF  THE  SCAMP  PRODUCTION  SYSTEM  WAS  WRITTEN. 
USING  THE  RULES  FOR  SIZING  BUFFERS  IN  A PRODUCTION  SYSTEM  AS  A GUIDE, 
NUMERICAL  SOLUTIONS  FOR  SEVERAL  ALTERNATIVE  OPERATING  MODES  WERE  DE- 
TERMINED. 

THE  RULES  CONCERNING  THE  PLACEMENT  OF  BUFFERS  ARE  OF  SUCH  A GENERAL 
NATURE  THAT  THEY  CAN  BE  APPLIED  TO  ANY  PRODUCTION  SYSTEM.  HOWEVER,  IN 
ORDER  TO  DETERMINE  NUMERICAL  VALUES,  A SPECIFIC  MATHEMATICAL  OR 
SIMULATION  MODEL  MUST  BE  AVAILABLE  FOR  USE. 

3.  BUFFER  RULES: 

THE  RULES  OBTAINED  FOR  THE  EFFECTIVE  PLACEMENT  OF  BUFFERS  WERE  DE- 
RIVED FROM  GENERAL  OBSERVATION  OF  THE  EFFECT  A BUFFER  WOULD  HAVE  ON 
THE  THROUGHPUT  OF  THE  PRODUCTION  SYSTEM.  THESE  RULES  ALSO  INCLUDE 
FACTS  THAT  ARE  TRUE  FOR  ANY  PRODUCTION  SYSTEM,  AND  CANNOT  NECESSARILY 
BE  CHANGED  BY  THE  ADDITION  OF  BUFFERS.  SOME  OF  THESE  RULES  MIGHT 
SEEM  ELEMENTARY,  BUT  THEIR  KNOWLEDGE  IS  REQUIRED  TO  INSURE  THAT  A 
COMPLETE  AND  COMPREHENSIVE  REVIEW  OF  THE  FACTORS  PERTAINING  TO  BUFFERS 
IN  A PRODUCTION  SYSTEM  IS  INCLUDED, 

THE  RULES  THAT  FOLLOW  ARE  BASED  ON  THE  SCAMP  SYSTEM  DIAGRAM  SHOWN 
IN  FIGURE  1. 

A.  A SYSTEM  WITH  NO  BUFFERS  WILL  YIELD  THE  MINIMUM  POSSIBLE  AVER- 
AGE THROUGHPUT  FOR  THE  SYSTEM. 


B.  THE  USE  OF  INFINITELY  LARGE  BUFFERS  WILL  YIELD  THE  MAXIMUM 
AVERAGE  THROUGHPUT  FOR  THE  SYSTEM. 

C.  WHEN  USING  BUFFERS,  THE  AVERAGE  THROUGHPUT  OF  A SYSTEM  CAN  BE 
NO  GREATER  THAN  THE  AVERAGE  THROUGHPUT  OF  THE  SLOWEST  SUBMODULE. 
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D.  THE  ADDITION  OF  BUFFERS  SMOOTHS  THE  PRODUCTION  PROCESS  BY 
REDUCING  THE  INTERACTION  OF  SUBMODULES  THAT  RESULT  WHEN  FAILURES 
OCCUR. 

E.  IF  THE  AVERAGE  THROUGHPUT  OF  THE  PRECEDING  SUBMODULE  IS 
LESS  THAN  THE  AVERAGE  THROUGHPUT  OF  THE  SUCCEEDING  SUBMODULE,  THE 
BUFFER  BETWEEN  THE  TWO  WILL  EVENTUALLY  BECOME  EMPTY. 

F.  THE  CLOSER  THE  BUFFERS  ARE  TO  THE  END  OF  THE  SYSTEM,  THE 
MORE  IMPORTANT  IT  IS  TO  INSURE  THAT  THEY  DO  NOT  EMPTY. 

G.  IF  A BUFFER  FILLS,  IT  WILL  CAUSE  THE  SHUTDOWN  OF  A MACHINE, 

BUT  IT  WILL  NOT  DECREASE  THE  AVERAGE  THROUGHPUT  FOR  THE  SYSTEM. 

H.  RUN  THE  SLOWEST  SUBMODULE  AT  FULL  SPEED  AT  ALL  TIMES  IN 
ORDER  TO  GET  MAXIMUM  PRODUCTION. 

I.  RUN  THE  LAST  SUBMODULE  IN  THE  SERIES  TO  ITS  MAXIMUM  CAPACITY 
IN  ORDER  TO  GET  THE  MAXIMUM  POSSIBLE  THROUGHPUT. 

4.  SCAMP  SIMULATION: 

THE  SIMULATION  OF  THE  SCAMP  SYSTEM  USED  IN  THIS  ANALYSIS  IS  A 
RELATIVELY  SIMPLE  PROGRAM  USING  A MONTE  CARLO  TECHNIQUE.  IT  IS  VERY 
GENERAL  IN  NATURE  SO  THAT  DIFFERENT  OPERATING  POLICIES  AND  DISTRIBU- 
TIONS  OF  TIME  TO  FAILURE  AND  REPAIR  CAN  BE  USED.  THE  PROGRAM  GENER- 
ATESATES  TIMES  TO  FAILURE  AND  REPAIR  BASED  ON  AN  EXPONENTIAL  DISTRIBU- 
TION. THE  QUANTITIES  IN  THE  BUFFERS  ARE  TABULATED  EACH  MINUTE  AND 
TESTED  TO  SEE  IF  THEY  HAVE  EMPTIED  OR  FILLED.  OPTIONS  ALSO  EXIST 
THAT  WILL  CHANGE  THE  OPERATING  SPEEDS  WHEN  BUFFERS  REACH  CERTAIN 
VALUES. 

5.  ALTERNATIVES  INVESTIGATED: 

NOW  THAT  THE  GROUND  RULES  FOR  THE  BUFFER  ANALYSIS  HAS  BEEN  ESTABLISHED, 
THE  DIFFERENT  OPERATING  POLICIES  THAT  NEED  TO  BE  ANALYZED  MUST  BE  DE- 
TERMINED. THESE  POLICIES  REPRESENT  FEASIBLE  ALTERNATIVES  THAT  CAN 
LEAD  TO  DIFFERENT  VALUES  FOR  THE  AVERAGE  THROUGHPUT.  IT  IS  NECESSARY 
TO  INVESTIGATE  THESE  POLICIES,  BECAUSE  OF  THE  NEWNESS  OF  THE  SYSTEM, 

AND  IN  ORDER  TO  DETERMINE  BUFFER  CHARACTERISTICS  FOR  EACH  OF  THE 
POLICIES. 

TWO  OPERATING  POLICIES  ARE  INVESTIGATED  AND  PRESENTED  IN  THIS  PAPER. 
FIRST,  THE  PRODUCTION  RATE  OF  THE  SUBMODULES  IN  THE  UP-MODE  WERE  MATCHED. 
THE  OPERATING  CONDITIONS  THAT  WERE  USED  WERE: 


CASE  SUBMODULE 

PRODUCTION  RATE 

io5o 

MTBF 

75 

MTTR 

25 

QUALITY 

98% 

PRIMER  SUBMODULE 

1080 

40 

8 

100% 

BULLET  SUBMODULE 

1080 

90 

30 

98% 

LOAD/ASSEMBLE  SUBMODULE 

1080 

40 

20 

100% 
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THE  SECOND  POLICY  THAT  IS  INVESTIGATED  HATCHES  THE  AVERAGE  THROUGH- 
PUT OF  THE  SUBMODULES,  THE  CONDITIONS  USED  WERE: 


CASE  SUBMODULE 

PRODUCTION  RATE 
1080 

MTBF 

75 

MTTR 

25 

QUALITY 

PRIMER  SUBMODULE 

952 

40 

8 

100%. 

BULLET  SUBMODULE 

1080 

90 

30 

98% 

LOAD/ASSEMBLE  SUBMODULE 

992 

40 

10 

100% 

SUBMODULE  SPEED  CHANGES  ARE  COMPARED  FOR  EACH  OF  THE  ABOVE  OPERA- 
TING POLICIES.  FIRST  SPEEDS  ARE  HELD  CONSTANT.  NEXT,  THE  SPEED  OF 
THE  PRECEDING  SUBMODULE  IS  INCREASED  WHEN  THE  BUFFER  LEVEL  DROPS  TO 
ONE  QUARTER  FULL  AND  IS  DECREASED  WHEN  THE  BUFFER  LEVEL  REACHES  THREE 
QUARTERS  FULL. 

FINALLY,  THE  OPERATING  SPEEDS  ARE  ONLY  INCREASED  AND  THEN  ONLY 
DECREASED  WHEN  THE  BUFFER  LEVELS  REACH  THE  ABOVE  MENTIONED  LEVELS. 

FOR  EACH  SPEED  CONDITION  AND  OPERATING  POLICIES,  TWO  BUFFER  LEVELS 
WERE  INVESTIGATED.  THESE  LEVELS  WERE  DETERMINED  USING  THE  MTTR  OF  THE 
PRECEDING  SUBMODULE.  THE  MAXIMUM  QUANTITY  ALLOWED  IN  THE  BUFFER  WAS 
THAT  QUANTITY  WHICH  WOULD  SUPPLY  THE  SUCCEEDING  SUBMODULE  WITH  INPUT 
FOR  TWICE  ITS  MTTR  OR  THREE  TIMES  ITS  MTTR. 

IN  ORDER  TO  VALIDATE  THE  POLICY  COMPARISONS,  THEY  MUST  BE  TESTED 
AGAINST  THE  SAME  CONDITIONS.  THIS  IS  DONE  BY  RE-INITIALIZING  THE 
RANDOM  NUMBER  GENERATOR  SO  THAT  THE  SAME  CONDITIONS  WILL  BE  RE-CREATED 
FOR  EACH  POLICY  THAT  IS  TESTED.  THE  DURATION  OF  ALL  SIMULATION  RUNS 
WERE  FOR  9600  MINUTES  OR  TWENTY  DAYS  OF  PRODUCTION  AND  BUFFERS  WERE 
INITIALIZED  TO  THE  HALF  FULL  LEVEL. 

THESE  OPERATING  POLICIES  REPRESENT  ONLY  A FEW  OF  THE  POSSIBLE  CON- 
DITIONS THAT  COULD  BE  INVESTIGATED,  THEREFORE,  EVERY  EFFORT  HAS  BEEN 
MADE  TO  GUARANTEE  THAT  THE  SELECTION  REPRESENTS  A GOOD  CROSS-SECTION 
OF  ANTICIPATED  OPERATING  ENVIRONMENTS. 

ALTHOUGH,  IT  MIGHT  NOT  SEEM  OBVIOUS,  THE  GENERAL  ANALYSIS  OF  BUFFERS 
HELPED  TO  LIMIT  THE  NUMBER  OF  ALTERNATIVES  THAT  NEEDED  TO  BE  INVESTI- 
GATED. THE  RULES  LIMITED  THE  SCOPE  OF  THE  ANALYSIS  AND  HELPED  PREVENT 
THE  INVESTIGATION  OF  UNFEASIBLE  ALTERNATIVES.  THE  STUDY  OF  THE  ALTERNA- 
TIVES ALSO  SUCCEEDED  IN  PRODUCING  ADDITIONAL  FACTS  PERTAINING  TO  THE 
EFFECTIVENESS  OF  BUFFERS  IN  A PRODUCTION  SYSTEM.  THESE  WILL  BE  LISTED 
IN  THE  RESULTS  SECTION  OF  THIS  PAPER. 

6.  results: 

SINCE  TWO  MAJOR  OPERATING  POLICIES  WERE  INVESTIGATED,  THE  RESULTS 
OF  EACH  SHALL  BE  PRESENTED  SEPARATELY. 

FIRST,  CONSIDER  THE  MATCHING  OF  PRODUCTION  RATES  IN  THE  UP-MODE. 

THE  CONTROLLED  CONFIGURATION  FOR  THE  FIRST  POLICY  CONSISTED  OF  AN  UN- 
BUFFERED PRODUCTION  SYSTEM  AND  ONE  WITH  INFINITE  CAPACITY  BUFFERS. 

FOR  THESE  CONFIGURATIONS,  IT  WAS  OBSERVED  THAT: 
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AVERAGE  THROUGHPUT  WITH  NO  BUFFERS  - 358,47/PIECES/MINUTE 

AVERAGE  THROUGHPUT  WITH  INFINITE  BUFFERS  - 887.40  PIECES/MINUTE 

THE  RESULTS  OF  THE  SIMULATION  WHEN  BUFFERS  WERE  SIZED  TWICE  THE 
MTTR  WERE: 

AVERAGE  THROUGHPUT  WITH  NO  SPEED  CHANGES  - 
AVERAGE  THROUGHPUT  WITH  INCREASE  W)  DE- 
CREASES IN  SPEED  - 

AVERAGE  THROUGHPUT  WITH  SPEED  INCREASES  ONLY 
AVERAGE  THROUGHPUT  WITH  SPEED  DECREASES  ONLY 

THE  POLICY  WAS  REPEATED  FOR  BUFFER  CAPACITIES  THAT  WERE  THREE  TIMES 
THE  MTTR: 


760.07/PIECES/MINUTE 

768 . 90/P I ECES /MINUTE 
-768.60/PIECES/MINUTE 
-737 . 66/P I ECES /MI NUTE 


AVERAGE  THROUGHPUT  WITH  NO  SPEED  - 
AVERAGE  THROUGHPUT  WITH  INCREASES  AND 
DECREASED  IN  SPEED  - 

AVERAGE  THROUGHPUT  WITH  INCREASES  ONLY  - 
AVERAGE  THROUGHPUT  WITH  DECREASES  ONLY  - 


795 . 6l/P I ECES /MI NUTE 

791.24/PIECES/MINUTE 
807 . 29/P I ECES /MINUTE 
783. 98/P I ECES /MI NUTE 


THE  CONTROLLED  CONFIGURATION  FOR  THE  POLICY  WHERE  THE  AVERAGE 
THROUGHPUTS  WERE  MATCHED  WAS  THE  SAME  AS  THE  FIRST  POLICY.  FOR  THIS 
CONFIGURATION, IT  WAS  OBSERVED  THAT: 

AVERAGE  THROUGHPUT  WITH  NO  BUFFERS  - 344.8  PIECES/MINUTE 

AVERAGE  THROUGHPUT  WITH  INFINITE  BUFFERS  - 790.29  PIECES/MINUTE 


WHEN  BUFFER  CAPACITIES  WERE  SIZED  TWICE  THE  MTTR: 

AVERAGE  THROUGHPUT  WITH  NO  SPEED  CHANGES  - 691.14 /PI ECES /MI NUTE 

AVERAGE  THROUGHPUT  WITH  INCREASES  AND 

DECREASED  IN  SPEED  - 703. 07 /PI ECES /MI NUTE 

AVERAGE  THROUGHPUT  WITH  SPEED  INCREASES  ONLY  - 737. 29/PI ECES /MI NUTE 
AVERAGE  THROUGHPUT  WITH  SPEED  DECREASES  ONLY  - 697.52/PIECES/MINUTE 

WHILE  RESULTS  OBTAINED  WITH  BUFFER  CAPACITIES  SIZED  THREE  TIMES 
THE  MTTR  WERE : 

AVERAGE  THROUGHPUT  WITH  NO  SPEED  CHANGES  - 
AVERAGE  THROUGHPUT  WITH  INCREASES  AND 
DECREASES  IN  SPEED  - 

AVERAGE  THROUGHPUT  WITH  SPEED  INCREASES  ONLY 
AVERAGE  THROUGHPUT  WITH  SPEED  DECREASES  ONLY 

FROM  THESE  RESULTS,  SOME  GENERAL  CONCLUSIONS  CAN  BE  DRAWN: 

A.  INCREASING  THE  OPERATING  SPEED  WHEN  THE  BUFFER  LEVEL  DROPS 
IS  THE  MOST  EFFICIENT  WAY  TO  INCREASE  THE  AVERAGE  THROUGHPUT  AND  IN- 
CREASE THE  EFFECTIVE  SIZE  OF  THE  BUFFERS. 


739 • 50/PI ECES/MINUTE 

746 . 35/P I ECES/MI NUTE 

- 753.47/PIECES/MINUTE 

- 722.70/PIECES/MINUTE 
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B.  DECREASING  THE  SPEED  WHEN  A BUFFER  STARTS  FILLING  DOES  NOT 
INCREASE  THE  AVERAGE  THROUGHPUT. 

C.  HATCHING  THE  PRODUCTION  RATES  IN  THE  UP-PjODE  GIVES  A HIGHER 
AVERAGE  THROUGHPUT  THAN  MATCHING  THE  AVERAGE  THROUGHPUT  RATE. 

D.  ON  THE  AVERAGE,  A BUFFER  CAPACITY  OF  TWICE  THE  MTTR  HAS  AN 
EFFICIENCY  OF  91%  COMPARED  TO  AN  INFINITELY  LARGE  BUFFER. 

E.  A BUFFER  CAPACITY  OF  THREE  TIMES  THE  MTTR  HAS  AN  EFFICIENCY 
OF  95%  COMPARED  TO  AN  INFINITELY  LARGE  BUFFER. 

BECAUSE  THE  SCAMP  SYSTEM  IS  STILL  UNDERGOING  DEVELOPMENTAL  TEST- 
ING, MUCH  MORE  WORK  ON  BUFFER  ANALYSIS  AND  MANY  MORE  OPERATING  POLI- 
CIES NEED  TO  BE  INVESTIGATED.  ALSO,  AS  SYSTEM  OPERATING  CHARACTERIS- 
TICS CHANGE  AND  COME  TO  LIGHT,  NEW  ANALYSES  SHOULD  BE  MADE  AND  HOPEFULLY, 
THIS  ANALYSIS  IS  OF  SUCH  A GENERAL  NATURE  THAT  ALL  NEW  CONDITIONS  CAN 
BE  ANALYZED  USING  THIS  TECHNIQUE. 


338 


STATISTICAL  MODEL  FOR  CONTROLLER  PERFORMANCE  MEASURES  FOR 
AN  AIR  TRAFFIC  AUTOMATED  SYSTEM  (ATMAC) 


Erwin  Biser 
Avionics  Laboratory 
U.  S,  Army  Electronics  Command 
Fort  Monmouth,  New  Jersey 


ABSTRACT 


The  Avionics  Laboratory  is  presently  engaged  in  the  construction  of 
a computer  queuing  model  to  evaluate  and  to  compare  candidate  system  con- 
figurations for  an  Air  Traffic  Automated  Center  (ATMAC). 

It  is  necessary  to  verify  that  the  model  generates  and  operates  on 
data  that  are  truly  representative  of  the  Air  Traffic  System  configura- 
tions being  simulated. 

The  ATMAC  queuing  model  simulates  Array  aircraft  operations,  the 
approach  control,  enroute  control  and  departure  control  functions  in  a 
division  area,  data  processing  and  communications  functions  in  regulating 
and  controlling  Army  Air  Traffic  consisting  of  100  to  300  aircraft  of 
various  types. 

In  order  to  validate  the  ATMAC  queuing  model,  an  Interactive  Simu- 
lator was  developed  to  integrate  the  various  data  processing  and  display 
equipment  and  to  integrate  the  software  program  packages  required  to  gen- 
erate the  desired  realistic  controller  responses.  The  data  recorded  and 
collected  from  the  Interactive  Simulator  include: 

Controller  time  response  data:  elapsed  time  from  displayed  alert 

to  acknowledge  action;  from  presentation  of  flight  blank  display  to  com- 
pleted flight  plan  entrance,  etc.  These  are  response  time  probability 
distributions  associated  with  each  of  the  sequence  of  tasks  performed  by 
an  operator  (controller) . 

The  enroute  control  function  for  instance,  includes  a measurement 
of  the  time  required  for  a controller  to  acknowledge  a conflict  alert. 

A scenario  Is  developed  to  verify  that  the  alert  acknowledged  time  measure- 
ments are  representative  of  realistic  (actual)  events.  Based  on  a designed 
realistic  scenario  discrete  action  required  by  the  Controller  are  deter- 
mined. From  the  distribution  of  operator  response  time  measurements,  time 
Intervals,  the  mean,  the  standard  deviations  and  the  cumulative  probability 
distribution  are  computed.  This  process  is  repeated  for  all  operator  task 
time  statistics  gathered  by  the  interactive  simulator. 
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In  addition  to  providing  valid  data  (from  the  actions  of  live  exper- 
ienced operators)  as  input  for  the  ATMAC  queuing  model,  the  interactive 
simulator  is  being  used  for  the  validation  of  the  ATMAC  queuing  model:  the 

data  collected  from  the  interactive  simulator  are  compared  with  the  results 
of  other  system  simulations.  These  data  include  controller  response  time 
distributions  associated  with  sequences  of  tasks  and  the  system  performance 
for  those  functions  incorporated  in  ..he  interactive  simulator.  The  system 
performance  data  consist  of:  number  of  flights  in  a sector,  number  of  con- 

flicts, number  of  near  misses,  separation  distance  between  any  two  planes, 
etc. 

The  basic  statistical  method  employed  in  the  analysis  of  the  inter- 
active simulator  data  is  that  of  a two-way  analysis  of  variance  to  analyze 
the  approach  control.  Departure  Control,  and  Handover  data.  It  identifies 
simultaneously  differences  in  operator  performance  between  individual  oper- 
ators and  between  levels  of  automation.  For  each  operator,  the  mean  re- 
sponse time,  the  number  of  measurements,  the  standard  error  of  the  mean, 
the  90  percent  confidence  interval  of  the  time  mean  are  printed  for  each 
task  and  for  each  mode  of  operation. 

> 

The  Scheffe  method  was  used  to-  compare  the  mean  response  times  of 
the  semi-automated  and  max-automated  modes  for  the  same  task.  The  data 
of  operators  which  consistently  yielded  response  times  significantly  dif- 
ferent from  the  majority  was  deleted  from  the  data  set.  The  analysis  was 
then  repeated  for  the  remaining  data.  Differences  in  the  response  times 
resulting  from  pairs  of  automation  levels  were  identified.  The  response 
times  from  different  levels  of  automation  represented  the  same  or  at 
least  similar  tasks. 

The  inputs  to  the  queuing  model  consist,  among  other  items,  of  the 
response  time  distribution  for  (one)  representative  operator  for  each  task, 
function  or  subfunction  of  the  ATMAC. 


1. 


INTRODUCTION 


The  basic  objective  of  the  Air  Traffic  Management  System 
is  to  facilitate  safe,  orderly , and  efficient  movement  of  air- 
craft throughout  the  combat  area  with  minimum  constraints  on 
mission  accomplishments. 

The  following  are  some  of  the  typical  missions  flown  by 
Army  rotary  wing  and  fixed  wing  aircraft. 

Tactical  airlift:  landing,  airdrop,  and  maximum  load 

Logistical  airlift:  troops,  normal  cargo,  and  maximum 

cargo 

Electronic  warfare 
Battlefield  illumination 
Search  and  rescue 
Medical  rescue 
Emergency  resupply 

Delivery  of  critical  personnel  and  supplies 
Radar  surveillance 
Infrared  surveillance 
Armed  reconnaissance,  etc. 

One  of  the  principal  objectives  of  ATMAC  is  to  accomplish 
safe  movement  of  aircraft  within  the  volume  of  responsibility. 
Each  aircraft  must  arrive  at  its  destination  safely  before  it 
can  carry  out  its  mission. 

Since  timeliness  is  an  important  factor  in  mission  accom- 
plishment, the  orderly  movement  of  aircraft  is  essential.  With 
an  orderly  movement  of  aircraft  in  the  terminal  area,  each  air- 
craft is  able  to  depart  the  airfield  close  to  the  pilot's  filed 
estimated  time  of  arrival. 

The  ATMAC  must  be  capable  of  handling  aircraft  operations 
24-hours  a day.  It  must  handle  high  traffic  densities  with 
minimum  traffic  delays. 

The  combat  area  represents  the  tactical  environment  for 
combat,  combat  support,  and  combat  support  units  that  are  di- 
rectly engaged  with  the  enemy. 
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2.  INTERACTIVE  SIMULATOR  (COMPUTER  PROGRAM)  DESCRIPTION 


The  Interactive  Simulator  (I/A  Sim)  is  a real-time  system  operating  under  the 
Varian  VORTEX  operating  system.  In  order  to  provide  validation  data  for  the 
ATMAC  Queuing  Model,  an  air  traffic  environment  is  generated  simulating  the 
movement  of  aircraft  and  performing  the  functions  of  console  input/output,  track 
state  maintenance,  maneuvering,  conflict  prediction  and  resolution,  and  approach/ 
departure  control. 

The  I/A  Sim  operates  under  the  Varian  VORTEX  operating  system  as  two 
multiprogrammed  tasks;  1)  an  event  processing  task  consisting  of  Scenario 
Input,  Track  State  Maintenance,  Approach/Departure  Control,  Conflict 
Prediction/Resolution,  Data  Recording  and  Reduction,  and  2)  a Console  Input/ 
Output  Task  consisting  of  Switch  Action  Processing  and  Display  Generation. 

Both  tasks  run  concurrently.  The  functions  in  the  event  processing  task  are 
called  in  from  rotating  memory  disc  storage  as  overlays  by  a system  scheduler 
program,  as  shown  in  the  following  diagram. 

The  i/A  Sim  has  three  basic  functions  in  common  with  the  Queuing  Model: 
Approach/ Departure  Control,  Track  State  Maintenance,  and  Conflict 
Prediction/Resolution.  Approach/Departure  Control  monitors  flights  within  a 
designated  radius  of  a terminal  checkpoint,  determines  patterns  of  approach  and 
departure,  and  calculates  spacing  of  aircraft  for  take-off  and  landing.  Track 
State  Maintenance  monitors  the  flight  enroute,  updates  its  position  and  heading, 
and  calculates  the  necessary  parameters  for  maneuvering.  Conflict 
Prediction/Resolution  monitors  flights  with  respect  to  other  flights  and  restricted 
areas.  When  a conflict  is  detected,  a display  alert  is  generated  for  the  console 
operator. 

Flights  and  fixed  points  are  initiated  through  the  Scenario  Input  Function. 

A preformatted  Scenario  input  file  is  read  and  data  is  stored  in  the  appropriate 
location  in  the  data  base.  This  data  is  then  processed  by  the  other  system 
functions,  thereby  generating  flights,  terminals,  boundaries,  and  navigational 

aids. 

The  five  display  consoles  of  the  I/A  Sim  are  controlled  by  the  Console 
Input/Output  function.  Updated  displays  are  output  to  each  console  at  periodic 
intervals.  All  input  from  the  alphanumeric  and  function  keyboards  and  joy/ 
pressure  stick  is  interpreted,  processed,  and  appropriate  data  is  stored  in  the 
data  base. 

The  following  paragraphs  describe  the  I/A  Sim  program  functions  in 
more  detail.  The  I/A  Sim  Program  Description  Document  (CDRL  item  G001) 
provides  a still  more  detailed  description  of  the  program. 

Console  I/O  - This  function  processes  inputs  from  the  display  consoles 
and  outputs  updated  displays  to  them.  This  function  is  core  resident  and  oper- 
ates every  200  Milliseconds.  It  performs  the  following  functions: 

• Output  updated  displays  to  the  console  or  consoles.  Whenever  dis- 
plays have  been  updated  (changed)  a flag  is  set  indicating  that  they 
are  to  be  output  to  the  display  consoles.  Displays  are  updated  when 
track  positions  change,  when  ARO’s  have  been  modified,  or  when 
new  categories  of  display  data  have  been  made  up  to  alert  the  con- 
sole operator  of  conditions  requiring  his  attention  or  in  response 

to  switch  actions  requesting  such  data, 

• Input  from  console  alphanumeric  keyboard.  Each  console  is  examined 
to  determine  whether  an  input  is  ready  (pending)  from  the  alpha- 
numeric keyboard.  If  so,  the  message  is  input,  buffered,  and  the 
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console  input  (switch  action)  processor  scheduled  to  operate  in 
the  next  cycle  of  the  i/A  Sim. 

• Input  from  the  function  keyboard.  Each  console  is  examined  to 
determine  whether  an  input  is  ready  (pending)  from  the  function  key- 
board. If  so,  the  input  is  read,  buffered,  and  the  console  input 
(switch  action)  processor  scheduled  to  operate  in  the  next  cycle  of 
the  I/A  Sim. 

• Input  from  the  joy/pressure  stick.  If  the  joy/pressure  stick  has  been 
activated  it  is  read  and  the  value  input  used  to  update  the  position 

of  the  hook  symbol  on  the  console  display.  Display  update  to  the 
console  from  which  the  input  was  received  is  set  to  cause  console 
I/O  to  output  displays  (including  the  updated  position  of  the  hook 
symbol)  to  that  particular  console  the  next  time  it  is  operated 
(200  milliseconds  later). 

System  Scheduler  — This  function  schedules  the  operation  of  the  pro- 
grams in  the  i/A  Sim.  This  function  is  core  resident  and  is  designated  as  a root 
segment  under  VORTEX,  The  functional  program  segments  of  the  I/A  Sim  are 
designated  as  overlay  segments  and  reside  on  disc.  'They  are  loaded  into 
memory  and  operated  on  a cyclic  basis  by  the  system  scheduler. 

Scenario  Input  Function  - This  function  is  read  into  memory  and  operated 
on  a cyclic  basis.  Its  task  is  to  read  scenario  input  data  (simulation  data)  from 
either  the  simulation  file  on  disc  storage  or  from  the  simulation  file  on  magnetic 
tape.  This  program  reads  such  data  into  memory  and  update  the  I/A  Sim  data 
base  (i.e.  tracking  tables,  etc.)  as  required. 

Track  State  Maintenance  — This  function  is  read  into  memory  and 
operated  on  a cyclic  basis.  Its  purpose  is  to  update  the  X,  Y,  Z,  velocity  and 
heading  for  each  active  track  in  the  i/A  Sim  system  depending  upon  the  parti- 
cular flight's  characteristics. 

Location  Monitor  — This  function  is  read  into  memory  and  operated  on  a 
multiple  cycle  basis  (i.e.  every  other  cycle,  every  third  cycle,  etc.).  Its 
task  is  to  monitor  maneuvering  flights  (i.e.  those  following  flight  plans)  and  to 
determine  at  what  point  they  should  be  required  to  maneuver  to  maintain  course 
along  a given  flight  path.  Predicted  maneuvers  are  communicated  to  the  dis- 
play update  function  for  the  making  up  of  displays  to  alert  the  console  operator 
of  suggested  course  changes. 

Track  Maneuvering  — This  function  is  called  when  it  is  desired  to 
maneuver  a track  in  the  i/A  Sim  system.  This  function  determines  the  neces- 
sary parameters  required  to  maneuver  a flight  to  another  heading,  another 
altitude,  another  speed,  or  to  a fixed  location  in  the  I/A  Sim  system.  The 
parameters  output  by  this  function  are  communicated  to  the  display  track  update 
function  for  formatting  displays  to  the  console  operator  indicating  the  required 
change  to  the  track.  Changes  to  tracks  are  communicated  to  the  ghost  pilot 
console  operator  who  then  implements  the  change  by  an  appropriate  switch 
action. 

Conflict  Prediction  - This  function  is  read  into  memory  and  operated  on 
a multiple  cyclic  basis  (i.e.  every  ether  cycle,  every  third  cycle,  etc. ),  Its 
function  is  to  monitor  the  position  of  all  flights  within  the  i/A  Sim  system  (a) 

In  respect  to  all  other  flights  and  (b)  in  respect  to  system  boundaries  and 
restricted  areas.  When  a conflict  is  detected,  such  data  is  communicated  to 
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the  display  update  function  for  the  making  up  of  displays  to  alert  the  console 
operator  of  the  condition.  The  conflict  prediction  algorithm  computes  suggested 
course  changes  for  display  to  the  console  operator  to  avoid  the  conflict. 

Approach/Departure  Control  - T)  e Approach/Departure  Control  routine 
is  entered  when  1)  an  approaching  flight  enters  within  a specified  radius  of  its 
terminal  checkpoint,  or  2)  whenever  a flight  is  scheduled  to  depart.  A/D  Con- 
trol monitors  the  flight,  determines  patterns  of  approach  and  departure,  calcu- 
lates spacing  of  aircraft  for  takeoff  and  landing,  and  provides  vectoring  instruc- 
tions for  formation  link  up  and  break  up. 

Display  Generation  — This  function  is  read  into  memory  and  operated  on 
a cyclic  basis,'  Its  purpose  is  to  convert  the  positional  data  for  the  tracks  in 
the  I/A  Sim  system  into  display  data  for  output  to  the  consoles.  In  addition, 
this  routine  makes  up  displays  for  a console  to  indicate  conflicts  and  suggested 
maneuvers  for  flights  in  the  system.  When  this  function  has  operated  the 
display  update  flag  is  set  indicating  that  the  Console  I/O  function  should  output 
updated  displays  to  the  consoles. 

Switch  Action  Processing”  This  routine  is  read  into  memory  and 
operated  only  upon  receipt  of  an  input  from  a console.  Its  function  is  to  pro- 
cess the  input  for  legality  and,  if  valid,  provide  the  proper  processing.  Func- 
tions performed  by  this  routine  include  the  following: 

1.  Format  display  for  output  to  console  (e,  g.  boundaries,  naviga- 
tional aids,  corridors,  etc.) 

2.  Input  or  modify  flight  plans 

3.  Change  track  speed,  altitude,  or  heading  (i.e,  vector  flight) 

4*  Accept  flight  from  outside  sector 

5.  Handover  flight  to  outside  sector 

6.  Handover  flight  to  tower  controller 

7.  Accept  flight  from  tower  controller 

8.  Break  up  or  link  flights 

9.  Change  display  scale  or  display  coordinates 

10,  Format  data  for  ARO  display 

11,  Initiate  or  drop  flights  within  system 

12,  Start  or  terminate  simulation  processing 

Once  switch  action  processing  has  been  completed  the  display  update 
flag  will  be  set  for  Console  I/O  indicating  the  updated  displays  are  to  be  output 
to  the  consoles. 

Data  Recording  — This  routine  is  read  into  memory  and  operated  on  a 
cyclic  basis  when  data  recording  has  been  requested.  Its  function  is  to  format 
data  for  output  to  the  recording  output  device. 

Data  Reduction  Program  — The  I/A  Sim  data  reduction  program  operates 
off-line  from  the  rest  of  the  I/A  Sim  program.  It  operates  upon  the  data 
collected  during  a simulation  and  computes  statistics  describing  operator  task 
time,s  and  parametric  function  effectiveness.  The  output  data  provided  by  this 
program  are  described  in  the  next  topic. 
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3.  TYPES  OF  DATA  PROVIDED  BY  INTERACTIVE  SIMULATOR 


The  I/A  Sim  collects  operator  response  time  and  algorithm  effectiveness  data  for 
all  major  ATMAC  controller  tasks  at  up  to  3 automation  levels. 


The  data  to  be  collected  from  the  i/A  SIM  for  validation  of  the  ATMAC 
Model  consists  of  controller  response  data  and  system  and  controller  perform- 
ance data.  Controller  data  includes  such  items  as  alert  response  time,  alert 
resolution  time,  flight  plan  entry/modification  time,  flight  plan  clearance  time, 
and  handover  coordination  time.  System  and  controller  performance  data  con- 
sists of  elapsed  time  of  the  simulation,  number  of  flights  in  a sector,  number 
of  conflicts,  near  misses,  restricted  zones  active,  penetrations  into  restricted 
zones,  separation  alerts,  arrivals,  departures,  flight  plans  files,  modified 
plans,  original  and  modified  flight  plan  errors,  plans  cleared,  plans  rejected, 
unassigned  flights,  and  late  handovers.  The  collected  data  is  output  after 
reduction  and  consists  of  cumulative  probability  distributions  including  means 
and  standard  deviations  for  all  time-dependent  categories  of  recorded  data. 

To  facilitate  the  collection  of  controller  response  data  with  the  I/A  SIM, 
validation  test  procedures  have  been  developed  for  each  of  the  ATMAC  functions 
implemented  on  the  I/A  SIM  as  follows: 

• Flight  Plan  Entry  and  Modification 

• Flight  Plan  Clearance 

• Departure  Control  Functions 

• Approach  Control  Functions 

• Enroute  Air  Traffic  Operation  Functions 

The  validation  test  procedures  are  contained  in  the  Validation  Test  Plan 
(CDRL  D001)  and  consist  of  two  parts.  The  first  part  is  a detailed  description 
of  the  actions  required  by  the  test  subject  controller  and  the  supporting  con- 
trollers, The  second  part  is  a series  of  flow  diagrams  which  parallel  the  first 
part  and  include  interactions  between  the  exercise  participants  and  the  I/A  SIM 
computer.  Since  most  of  the  I/A  SIM  functions  are  being  evaluated  for  more 
than  one  automation  level,  test  procedures  are  included  for  each  level  as 
required. 

The  automation  levels  provided  in  the  I/A  SIM  are  as  follows: 

• Flight  Plan  Entry  — Semi -automated 

• Flight  Plan  Modification  - Semi-automated 

• Flight  Plan  Clearance  - Semi -automated 

• Departure  Control  Operations  — Min.  Auto  and  Max.  Auto 

• Approach  Control  Operations  — Min.  Auto  and  Max.  Auto 

• Handover  Operations  —Min.  Auto  and  Max.  Auto 

• Conflict  Prediction  - Automated 

• Conflict  Resolution  - Min.  Auto,  Semi-automated  and 


Max.  Automated 

• Flight  Plan  Monitor  - Min.  Auto  and  Max.  Automated 

The  types  of  operator  tasks  involved  in  the  various  modes  of  the  I/A  SIM 
listed  above  Include  the  following: 

Conflict  Prediction/ Resolution  Data  - Data  collected  for  conflict  predic- 
tion and  resolution  include  controller  response  times  for  conflict  alert  acknowledge, 
conflict  resolution  determination,  conflict  resolution  coordination,  and  flight 
vectoring.  In  addition,  the  program  measures  the  effectiveness  of  the  conflict 
resolution  decisions  and  the  effectiveness  of  aircraft  separation  performed  by 
the  controller. 
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Approach/ Departure  Control  Data  - Data  measurements  collected  for 
approach/departure  control  include  controller  participation  time  and  effective- 
ness of  control  for  flight  breakup  and  flight  linkup.  Flight  breakup  is  an 
approach  control  task  where  an  incoming  flight  of  several  aircraft  is  divided 
into  Individual  aircraft  tracks  to  permit  final  approach  and  landing.  Flight  linkup 
is  a departure  control  task  which  combines  aircraft  which  depart  separately- 
into  one  flight.  Data  measurements  also  are  made  for  the  approach  control 
tasks  associated  with  aircraft  vectoring,  sequencing  and  metering.  Both  con- 
troller times  and  control  effectiveness  will  be  recorded  for  these  tasks. 

Flight  Plan  Entry/Modification  Data  - Controller  time  required  to  enter 
and  update  flight  plans  are  collected  for  this  function.  For  this  operation,  the 
controller  enters  flight  plan  data  with  a keyboard  entry  device.  The  computer 
records  entry  time  and  also  measures  accuracy  of  data  entry. 

Flight  Plan  Clearance  Data  - Controller  time  required  to  perform  flight 
plan  clearance  is  collected  for  this  function.  Flight  plan  clearance  involves 
examining  the  plan  parameters  and  comparing  them  with  other  cleared  flights 
for  possible  conflicts. 

Handover  Data  - Data  collected  for  handover  situations  include  controller 
response  times  for  handover  alert  acknowledgment  and  handover  resolution. 

These  data  Items  will  be  collected  for  the  following  handover  situations: 

• Enroute  to  enroute 

• Enroute  to  approach 

• Approach  to  tower 

• Approach  to  GCA 

• Approach  to  TLS 

• Tower  to  departure 

The  general  form  of  the  output  data  consists  of  horizontal  cumulative 
probability  distributions  with  units  on  the  top  line  with  the  cumulative  percents 
below,  as  shown  in  the  table  on  the  facing  page.  Non-time -dependent  data  is 
also  output  horizontally;  for  example:  Number  of  Flights  - 54.  The  figure  on 
the  opposite  page  is  an  example  of  the  output  for  enroute  conflict  resolution. 

In  the  figure,  d represents  a decimal  digit.  Detailed  descriptions  of  the  I/A 
SIM  output  formats  are  contained  in  Vol.  IV  of  the  I/A  SIM  Program  Descrip- 
tion Document. 
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Table  I.  EXAMPLE  OF  THE  OUTPUT  FORMATS  PROVIDED  BY  THE  I/A  SIM  COMPUTER  PROGRAM 

ENROUTE  CONFLICT  RESOLUTION  MEASURES 
AIR -AIR 

ACKNOWLEDGE  ALERT  MEAN  =7,5  ST ANDARD  DEVIATION  = 2.7 

1 2 3 4 5 6 7 8 9 10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  Seconds 

00  03  09  15  25  36  47  51  60  65  72  80  83  90  92  93  94  96  96  98  99  00  00  00  00  00  00  00  00  Percent 

RESOLVE  CONFLICT  MEAN  = 23. 1 STANDARD  DEVIATION  = 5.  0 

5 10  15  20  25  30  35  40  45  50  55  60  65  70  75  80  85  90  Seconds 

00  00  25  49  65  75  84  90  91  95  98  98  98  98  00  00  00  00  Percent 

NUMBER  OF  FLIGHTS  = 54  CONFLICTS  = 18  NEAR  MISS  = 5 ELAPSED  TIME  = 185  MIN 

RESTRICTED  ZONES 

ACKNOWLEDGE  ALERT  MEAN  = d.d  STANDARD  DEVIATION  = d.d  SEC 

1 2 3 4 5 6 7 8 9 10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  Seconds 

dd  dd  dd  dd  dd  dd  dd  dd  dd  dd  dd  dd  dd  dd  dd  dd  dd  dd  dd  dd  dd  dd  dd  dd  dd  dd  dd  dd  dd  Percent 

RESOLVE  CONFLICT  MEAN  = dd.d  STANDARD  DEVIATION  = d.  d SEC. 

5 10  15  20  25  30  35  40  45  50  55  60  65  70  75  80  85  90  Seconds 

dd  dd  dd  dd  dd  dd  dd  dd  dd  dd  dd  dd  dd  dd  dd  dd  dd  dd  Percent 

NUMBER  OF  ZONES  = dd  CONFLICTS  = dd  PENETRATIONS  = dd  ELAPSED  TIME  = dd  MIN 


The  two  primary  activities  performed  by  the  ATMAC  Queuing  Model  are 
the  functional  application  of  air  traffic  control  algorithms  and  the  determination 
of  system  events  in  simulated  time  based  upon  these  algorithms.  The  functional 
simulations  respond  to  the  status  of  system  conditions  and  determine  that  a 
sequence  of  simulated  system  activity  is  required  (e.g.  t a controller  must 
vector  a flight  in  response  to  a conflict  alert).  This  need  is  communicated  to 
ASP  which  then  initiates  and  processes  a Table  of  Action  Sequences  and  Schedules 
(TASS),  modeling  the  desired  chain  of  simulated  events.  ASP  simulates  the 
utilization  of  controllers  and  system  hardware  such  as  the  central  data  processor, 
(computer),  G/G  DDL  and  G/A/G  DDL  facilities.  The  TASS  contains  control  and 
instruction  parameters  relating  to  this  simulation  capability  as  discussed  above. 
Data  concerning  the  utilization  of  system  resources  (controllers,  data  processor 
and  DDL)  are  collected  by  ASP  and  are  stored  for  later  reduction  by  the  Model 
Data  Reduction  Program. 

A single  TASS  may  be  in  process  for  more  than  one  flight  during  the  same 
simulated  time  period.  In  cases  where  two  or  more  flights  are  assigned  to  the 
same  controller  and  have  the  same  TASS  in  process,  ASP  provides  the  queuing 
priority  and  maintains  the  queued  tasks  until  the  controller  becomes  free  to 
handle  the  next  task. 

ASP,  residing  at  the  center  of  these  activities,  provides  the  primary 
model  control  mechanism.  As  such  it  receives  and  processes  requests  from 
external  sources  (functional  modules)  to  initiate  a sequence  of  TASS  processing 
steps  and  from  internal  sources  such  as  requests  encountered  through  processing 
TASS  parameters.  The  external  source  interface  is  through  the  GASP  event 
chain.  Consider  the  example  of  the  ASP  interface  processing  which  occurs  when 
the  Conflict  Prediction  algorithm  determines  that  a flight  is  in  conflict  with  a 
restricted  zone,  as  described  in  the  preceding  sample  TASS.  The  Air-Air  Con- 
flict Prediction/Resolution  TASS  must  be  processed  to  simulate  the  controller’s 
efforts  in  attempting  to  resolve  the  conflict  in  accordance  with  the  sequence  of 
events  described  in  the  next  paragraph. 

The  Conflict  Prediction  algorithm  passes  the  appropriate  parameter 
data  (the  flight’s  identification  and  control  parameters  which  request  ASP  pro- 
cessing) to  GASP.  The  GASP  routines  store  this  data  in  the  GASP  event  chain. 
When  the  Conflict  Prediction  algorithm  is  finished  testing  for  conflicts,  it 
returns  control  to  GASP.  As  GASP  processes  the  event  chain,  it  removes  the 
next  event  in  time  sequence  from  the  event  chain.  The  control  parameters 
placed  In  the  event  chain  by  the  Conflict  Prediction  algorithm  are  interpreted  as 
a request  for  ASP  processing  and  control  is  passed  to  ASP.  ASP  receives  control 
through  Its  external  source  interface  using  the  control  parameters  associated 
with  the  event  to  initiate  the  interpretation  of  the  appropriate  TASS.  Processing 
the  TASS  request  begins  immediately  and  the  internal  source  interface  is 
exercised.  Some  requests  such  as  branches  and  ASP  services  are  performed 
wholely  within  the  ASP  module.  Other  requests,  such  as  simulating  usage  of  a 
controller,  cause  ASP  to  place  parameters  into  the  GASP  event  chain  which 
request  future  ASP  processing. 

. ‘During  ASP  processing  of  the  Conflict  Prediction/Resolution  TASS  two 
requests  for  execution  of  functional  application  algorithms  are  encountered. 

The  ASP  processing  is  the  same  for  both  even  though  the  control  parameters 
are  different.  ASP  places  the  control  parameters  into  the  GASP  event  chain 
and  continues  processing  the  TASS.  After  the  next  time  delay  event  is  pro- 
cessed ASP  returns  control  to  GASP.  GASP  again  processes  the  event  chain 
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removing  the  next  event  in  time  sequence  from  the  chain.  The  control  param- 
eters placed  in  the  event  by  ASP  are  interpreted  to  be  a request  for  one  of  the 
two  prediction/resolution  functional  applications  and  GASP  passes  control  to  the 
requested  routine.  After  operating  on  the  data  base,  the  application  algorithm 
returns  control  to  GASP,  completing  the  sequence  of  simulated  activities  caused 
by  the  detection  of  the  conflict  situation. 


Queuing  Model  Block  Diagram:  TASS  Interface  Relationship 
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4. 


OUTPUT  DATA  PROVIDED  BY  QUEUING  MODEL 


Output  data  from  the  queuing  model  provides  information  for  performing  analysis  of 
ATMAC  system  effectiveness  by  providing  data  on  the  most  significant  parameters 
which  have  sensitivity  to  scenario  factors  and  to  differences  between  the  ATMAC 
configurations  being  modeled* 

The  queuing  model  output  and  reduction  functions  provide  data  for  analy- 
sis of  the  system  effectiveness  measures  of  an  ATMAC  configuration.  These 
system  effectiveness  measures  provide  the  means  for  comparison  of  alternate 
ATMAC  configurations. 

During  the  execution  of  the  queuing  model,  performance  data  is  continu- 
ously written  to  disk  for  post-processing.  This  performance  data  consists  of 
events  that  are  about  to  occur  or  have  just  been  completed,  the  time  of  day,  and 
other  information  about  the  event. 

After  the  execution  of  the  queuing  model  simulation,  offline  processing 
and  reduction  of  the  performance  data  begins.  The  performance  data  is  read 
into  the  Varian  computer  from  disk  and  is  processed  by  a postprocessing  pro- 
gram. This  program  groups  the  data  into  many  different  sequences  to  provide 
the  required  data  reduction  and  concludes  by  printing  data  reduction  reports. 

The  reduction  reports  consist  of  performance  measures  for  the  preceding 
simulation  of  a specific  ATMAC  configuration.  They  consist  of  Arrival /Depar- 
ture rates,  conflict  rates,  conflict  resolution  effectiveness,  flight  plan  monitor 
effectiveness,  appro ach/departu re  control  effectiveness,  flight/aircraft  densi- 
ties, aircraft/flight  delays  and  thruput,  and  processor,  digital  data  link,  and 
controller  utilization.  The  type  of  information  printed  for  each  of  these  cate- 
gories consists  of  "time-line’'  reports  showing  the  actual  time  sequences  in 
which  events  occurred,  mean  and  standard  deviations  of  event  occurrence  times 
and  a tabular  report  of  the  event  giving  a percentage  of  occurrence  (i.e. , a dis- 
tribution) of  each  type  of  category.  Accumulations  of  the  occurrence  of  the  total 
time  required  for  an  event  in  each  category  are  also  printed.  A detailed  des- 
cription of  the  queuing  model  output  formats  is  given  in  Vol.  V of  the  ATMAC 
Model  Program  Description  Document. 

I3y  relating  these  reports  of  performance  measures  for  each  ATMAC  con- 
figuration simulated  it  is  possible  to  evaluate  ATMAC  system  performance 
among  the  candidate  configurations. 
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5.  PLAN  FOR  ANALYSIS  OF  INTERACTIVE  SIMULATOR  RESULTS 


Each  of  the  ATMAC  functions  simulated  on  the  Interactive  Simulator  generates 
data  which  represent  controller  response  times,  switch  actions  and  efficiency 
of  operation. 

To  insure  that  these  performance  measures  are  representative  and  can  be  used 
on  the  ATMAC  Queueing  Model,  the  following  preparation  and  analyses  will  be 
performed. 

• Prepare  Plan  of  Experiments 

• Statistical  Analysis  of  Results 

• Statistically  Compare  Results  with  Previous  Studies 


6,  PLAN  OF  EXPERIMENTS 

The  Plan  of  Experiments  defines  a series  of  tests  which  are  designed  to  gather 
human  performance  data  for  ATMAC  functions  implemented  in  the  i/A  SIM.  The 
personnel  involved  in  the  tests  will  include  test  subjects,  test  conductors,  ghost 
pilots  and  ghost  controllers.  Test  subjects  with  sufficient  experience  to  be  clas- 
sified as  Air  Traffic  Controllers  will  be  used.  Each  test  subject  will  be  made 
familiar  with  the  I/A  SIM  and  the  ATMAC  functions  through  a training  program 
prior  to  collection  of  test  data. 

The  experiments  will  present  the  test  subject  with  air  situations  based  on 
approved  scenarios.  The  test  subject  and  the  support  personnel  (test  condia.  r, 
ghost,  controllers)  will  perform  the  experiment  using  the  procedures  described. 


7.  STATISTICAL  ANALYSIS  OF  RESULTS 


The  Plan  of  Experiments  test  program  has  been  designed  for  eight  subjects  who 
are  assumed  to  have  essentially  the  same  level  of  skill  in  air  traffic  control. 
Precautions  shall  be  taken,  however,  to  avoid  the  contaminating  effects  of  dif- 
ferent experience  levels. 


V?1 


in  order  to  prevent  biasing  the  experimental  response  data  by  individual 
differences  of  skill,  a training  session  of  at  least  two  days  with  appropriate  test- 
ing procedures  will  be  used. 

The  focus  of  data  evaluation  shall  center  about  the  purpose  of  the  I/A  Simulator 
and  the  ATMAC  Model.  The  primary  goal  is  to  gather  meaningful  human  response 
data  which  can  be  applied  to  the  ATMAC  Queuing  Model. 

The  principal  statistical  methods  which  will  be  employed  to  analyze  the  data  are 
an  analysis  of  variance  for  a two-way  classification  together  with  the  standard 
Tukey  and  Scheffe'  methods  (denoted  T-method  and  S-method,  respectively)  for 
multiple  comparisons.  The  analysis  of  variance  model  proposed  in  the  follow- 
ing section  results  in  the  standard  F-test  for  accepting  or  rejecting  hypotheses 
concerning  the  mean  response  times  for  the  three  different  levels  of  automation. 


8.  EXPERIMENTAL  DESIGN* 


This  section  describes  briefly  the  two-way  classification  model  for  the  analysis 
of  variance  to  be  performed.  It  is  assumed  that  the  response  time  of  each  oper- 
ator for  each  class  of  tasks  under  each  level  of  automation  will  be  sampled 
times.  The  sampled  response  times  will  be  denoted  X ^ where  i = 1,2, ..  .n  1J 
denotes  the  operator,  j = 1,2, ,.  .m  denotes  the  level  of  automation,  and  k - 1, 

2, . . . Ny  denotes  the  sample  size. 

The  basic  assumption  for  the  subsequent  analysis  is  that  the  random  variable 
Xyk  may  be  represented  as  follows  (see  H.  Scheffe':  The  Analysis  of  Variance, 
Wiley  and  Sons,  1959): 


( xijk~  M+Dti+/V7ij  + Eijk’ 


QX 


n m 

j - O,  = O, 

i-1  1 j=l  3 


2yo-Srir°- 


1=1  - IJ 


EJjk  is  N(0,<t  ). 


In  the  future,  the  foregoing  set  of  assumptions  will  be  referenced  as  hypothesis 
Q.  The  components  of  are  described  in. the  following  li«*t: 

li  = general  mean, 

otj  = variation  in  the  mean  due  to  operator  i, 

jS  j - variation  in  the  mean  due  to  the  level  j of  automation, 

*The  Analysis  of  Variance,  H.  Scheffe,  J.  Wiley  & Sons, 

1959. 
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- interaction  between  operator  i and  automation  level 

2 

= random  sample  error  from  a N(0,cr  ) distribution. 


Note  that  the  expected  value,  E(Xyk),  of  X|jk  is,  therefore, 


E<V 


= p + a.  + B . +7., 

i j ij 


In  order* to  be  specific,  /?  1 will  represent  the  variation  in  the  mean  due  to  the 
min- automated  mode  of  operation,  ft  2 the  variation  due  to  the  semi -automated 
mode  of  operation,  and  $3  the  variation  due  to  the  max-automated  mode  of 
operation. 

There  are  three  separate  hypotheses  which  will  be  of  interest  in  analyzing  the 
experimental  data;  they  are: 

H0:rij  K 0 for  i=1»2>  * • *n  and 

J = 1,2, ...m, 

- O for  j=l,2, . . .m, 

Hgiotj  “ O for  i-1, 2, . . . n. 


The  hypothesis  H0  is  of  no  direct  interest;  however,  acceptance  or  rejection  of 
H0  determines  the  specific  statistics  used  to  test  Hi  and  H2  and  influences  sig- 
nificantly any  interpretation  of  the  results  of  those  tests.  In  particular,  if  the 
hypothesis  H0  is  accepted,  by  statistical  inference  or  otherwise,  then  inferences 
about  the^i's  and  the/3  jfs  will  be  sufficient  to  summarize  the  experiment.  Thus, 
if  it  is  concluded  that /3  3 < /3 1,  then  the  difference  in  the  response  times  between' 
levels  of  automation  is  the  same  for  each  operator.  If  H0  is  rejected,  however, 
then  £3  < pi  implies  only  that  operator  response,  when  averaged  over  all  oper- 
ators, is  greater  in  the  min-auto  mode  than  in  the  max-automated  mode  of 
operation. 

The  important  hypothesis  with  respect  to  the  Interactive  Simulator  is  Hi,  the 
alternative  hypothesis  for  Hi.  Note  that  rejection  of  Hi  automatically  implies 
acceptance  of  Hi.  In  order  to  facilitate  the  computations,  Hi  will  be  tested 
rather  than  testing  Hi  directly.  If  H is  rejected,  then  the  T-method  will  be 
used  to  identify  significant  differences  between  the  /3jrs  under  the  hypothesis  II0. 

If  H0  is  rejected  (that  is,  interactions  are  present),  the  T-method  cannot  be 
used  because  the  j 1 s may  not  have  equal  variances. 

Hypotheses  Ho  is  included  in  order  to  verify  that  the  operators  have,  at  least 
approximately,  the  same  level  of  skill.  If  the  hypothesis  H2  is  rejected,  the 
T~method  (or  the  S-mcthod  if  H0  is  rejected)  will  be  used  to  determine  which 
operator  (or  operators)  contributed  to  the  rejection.  The  analysis  then  can  be 
repeated  after  deletion  of  the  relevant  operator  data  or  the  experiments  can  be 
repeated  by  the  appropriate  operator  (or  operators)  after  the  appropriate  training. 


9 . COMPUTATIONAL  PROCEDURE 


In  this  section  the  statistics  which  are  necessary  to  test  the  hypotheses  outlined 
in  the  previous  section  are  defined.  Recall  that  represents  the  sample 
response  time  of  operator  i in  the  operating  mode  j . 

Assume  that  the  data  set  Xijk  is  given.  In  order  to  test  the  hypotheses  men- 
tioned previously,  it  will  be  necessary  to  compute  the  following  sample  means 
and  estimates  (denoted  collectively  as  the  Means): 


1_ 

k-1 
n m 


X ij  * Ny  Z xijk’ 


Nt  = £ £ 

* i=l  j=l  13 

m ^ m 


b=l 


n *11 

hj  =5  x v=SxijNii* 

k=  I 
m 

G’j  " ]C  N..,  ; row  sums  of  cell  numbers 

1 j=1  fJ 


m 


Ai  “ m ^ Xij ' 


H.  — N..f  ; column  sums  of  cell  numbers 
J i=l  1J 


1 

Bi  S xij- 


Following  Scheffe1:  The  Analysis  of  Variance,  the  maximum  likeli- 
hood estimates  of/Jj,  denoted  by/?j,  are  the  solutions  of  the  following  set  of 
linear  equations: 


^ Rj,  j 1,2,.. .m. 


35^ 


where 


wK¥)- 


The  sums  of  squares  (denoted  collectively  by  SS)  which  are  necessary  for  the 
analysis  of  variance  are: 

* m Ni1  - m /h?\ 


n m MJ  , » \ 

^ 6 W- 

Tlr„  n2 

i n p ® Jk 

^S'^a-Lw.  £ vviAi  • 

i=l  [i=l  J Li=l  J 


ss==  £ 


N„ 

m n il 


}=i 1=1 


^ ^ gi 

v ■ S °i  ’ 


rm 

-1  r-  -.2 

m 1 

£ u 

fe  °< 

U=i  3J 

SSC  = 


n m ^ij  2 ^ v3'  2 

= 53  £ 53  <xiik  " Xij*  ' ^ ^ 

M j=i  ril  J 3 >1  j=i  J J 


where  the  weighting  factors  and  are  defined: 

<*»  ^ 
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Table  II . ANALYSE  OF  VARIANCE  FOR  A TWO-WAY  CLASSIFICATION*  SAMPLE  MEANS 


Table  III.  ANALYSIS  OF  VARIANCE  FOR  A TWO-WAY  CLASSIFICATION;  SUMS  OF  SQUARES 
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Table  XV.  QUEUING  MODEL  AUTOMATION  LEVELS 


Manual 

Level* 

■ 

Min-Automated 

Level 

Semi-Automated 

Level 

Max-Automated 

Level 

Facility 

Computer 

No 

Yes 

Yes 

Yes 

G-G  DDL 

No 

No 

Yes 

Yes 

G-A  DDL** 

No 

No 

No 

Yes 

A-G  DDL 

No 

► 

Yes 

Yes 

Yes 

Function 

Track  Acquisition/Tracking 

CONT 

DP 

DP 

DP 

FP"  Processing/Approval 

CONT 

DP** 

DP 

DP 

FP  Conformance 

CONT 

CONT 

DP 

DP 

Approach/Departure  Control 

CONT 

CONT 

DP 

DP 

Formation  Linkup/Breakup 

CONT 

CONT 

DP 

DP 

Handover 

CONT 

CONT 

DP 

DP 

Conflict  Prediction/Resol. 

CONT 

CONT 

DP 

DP 

♦Uses  radar  video 
♦♦Controller  approves  FP 

NOTE:  CONT  means  controller  performs  function 

DP  means  data  processor  performs  function  with  controller  review, 
approval  and/or  override. 
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When  these  computations  have  been  completed,  the  hypotheses  H0,  H^,  and  H2 
will  be  tested  by  the  procedures  enumerated  below: 

Step  1:  Test  H0  against  Q at  the  significance  level  a-  (=  0. 05).  In  order  to  perform 
this  test,  define  the  statistic: 

Nt  - mn  SSC  - SS£ 

F = (m-l)(n-l)  SS^ 

The  random  variable  F has  an  F-distribution  with  parameters  (m-l)(n-l)  and 
(Nt-mn).  The  F-test  for  H0  is  defined  as  follows:  Reject  H0  if  F > F0  where  F0 
satisfies: 

P(f  < F^)  = 1 -a. 

Step  2:  Test  Hi.  If  H0  is  accepted,  test  Hi  against  ft  HQ;  that  is,  set: 

Nfc  - n - m + 1 SSR  - SSC 
F=  tin  SS^  • 

The  random  variable  F has  an  F-distribution  with  parameters  (m-1)  and 
(Nt-m-n+1).  An  F-test  similar  to  that  of  Step  1 may  be  used  to  accept  or  reject 
Hi. 


If  H0  is  rejected,  then  Hi  should  be  tested  against  Q rather  than  ft  n Q;  the 
appropriate  statistic  in  this  case  is: 

N.  - mn  SSL 

TP  — t . SL 

* m-1  SS^, 


which  has  an  F distribution  with  parameters  (m-1)  and  (Nt-mn). 

As  noted  previously,  rejection  of  Hi  is  equivalent  to  acceptance  of  the  hypothesis 
that  there  are  significant  differences  in  the  /3jfs,  that  is,  differences  other  than 
those  produced  by  sampling  errors.  The  significance  of  accepting  or  rejecting 
H0  upon  Hi  was  discussed  in  the  preceding  section.  If  H0  is  accepted,  the 
T-method  will  be  used  to  determine  the  significant  differences  in  the  £j*s  when- 
ever Hi  is  rejected.  If  H0  is  rejected,  then  the  S-method  will  be  used  rather 
thftTi  the  T-method  to  determine  significant  differences  in  the  /3  j f s when  H^  is 
rejected. 


Step  3:  Test  H2.  If  H0  is  accepted,  test  H2  against  ft  0 Q with  the  statistic: 


F = 


N. 


m - n + 1 
~n^l 


C 


which  has  an  F distribution  with  parameters  (n-1)  and  (N^-m-m-l).  The  standard 
F-test  will  be  employed  (see  Step  1). 
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If  H0  is  rejected,  H2  should  be  tested  against  Q.  In  this  case, 

N.  - mn  SSA 
* n-1  SSE 

Acceptance  of  H2  is  equivalent  to  acceptance  of  the  hypothesis  that  all  of  the 
operators  have  an  equal  level  of  skill.  Conversely,  rejection  of  H2  implies  that 
the  operators  have  unequal  levels  of  skill;  in  this  case,  the  S-method  or  the  • 
T-method  (as  above)  will  be  used  to  determine  which  operators  were  respon- 
sible for  rejecting  H2. 

10 . EXPECTED  DATA  RESULTS 

All  communications  shall  be  recorded  for  detailed  analysis.  Operator  workloads 
shall  be  recorded  throughout  the  entire  run.  The  operation  workload  for  the  pur- 
pose of  this  study  is  defined  as  the  sum  of  all  aircraft  being  displayed  or  under 
his  control  plus  all  communications,  verbal  or  otherwise,  that  he  is  engaged  in 
at  any  discrete  period  of  time. 

Data  analysis  of  cumulative  probability  distributions,  summary  totals,  and 
flights  per  time  displayed  coupled  with  statistical  test  applications  will  provide 
knowledge  of  differences  in  human  operator  responses  which  will  be  attributable 
to  mode  manipulations  i.e.,  max-automated  or  min- automated  rather  than 
sampling  error. 

The  results  of  this  comprehensive  data  collection  and  analysis  will  provide  inputs 
to  the  ATMAC  Model  which  will,  within  the  limitations  of  the  I/A  Simulator,  be 
both  realistic  and  meaningful. 

The  data  to  be  gathered  for  each  test  subject  and  the  treatment  of  the  data  for 
each  ATMAC  function  to  be  examined  on  the  I/A  SIM  are  contained  in  the  follow- 
ing sections, 

a.  Flight  Plan  Entry/Modif icat ion/ Clearance . 


The  following  calculations  will  be  made. 


Means 

SD 

C.  P.D. 

Flight  plan  entry  time 

XX 

XX 

XX 

Flight  plan  modification  time 

XX 

XX 

XX 

Flight  plan  clear- reject  time 

XX 

XX 

XX 

The  data  output  i.e. , number  of  flight  plans  entered,  etc.  is  expected  to  be  large. 
For  this  reason  no  unusual  statistical  test  of  significance  is  envisioned;  however, 
if  the  sample  size  should  diminish  or  be  affected  by  widely  discrepant  subject 


360 


responses,  the  Student  ”t,f  test  shall  be  employed  between  subject  means. 
Cumulative  probability  distributions  and  sample  size  shall  determine  the  need 
for  the  use  of  the  Student  ,Tt". 


b . Approach  Control. 


The  following  calculations  shall  be  made  for  each  mode,  max-automated  and 
min-automated. 


Means 

SD 

C.  P.D. 

Accept  hand-in  time 

XX 

XX 

XX 

Approved  vector  instructions  time 

XX 

XX 

XX 

Resolve  separation  alert  time 

XX 

XX 

XX 

Closest  approach  distances 

XX 

1 — 

XX 

XX 

Data  shall  be  compiled  for  each  subject.  A total  mean  shall  be  calculated  for 
each  measure  in  each  mode.  The  hypothesis  to  be  tested  shall  be: 


Hr /3r/32^° 

The  implied  null  hypothesis  is: 

Hl:/31=  ^2  = 0 

The  significance  level  shall  be  set  at  0. 05. 

The  F-test  shall  be  used  to  test  the  differences  between  sample  means.  In  spe- 
cific terms  the  research  hypothesis  is  that  the  mean  responses  in  the  min- 
automated  mode  will  be  different  than  the  mean  responses  in  the  max-automated 
mode  and  that  such  a difference  is  not  due  to  sampling  error. 


c . Departure  Control. 

The  following  calculations  shall  be  made  for  each  mode,  max-automated  and 
min-automated. 


Means 

SD 

C.P.D. 

Departure  alert  acknowledge  time 

XX 

XX 

XX 

Approve  vector  instructions  time 

- 

XX 

XX 

Resolve  Separation  alert  time 

XX 

XX 

XX 

Closest  approach  distances 

XX 

XX 

XX 
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Data  shall  be  compiled  for  each  subject.  A total  mean  shall  be  calculated  for 
each  measure  in  each  mode.  The  hypothesis  to  be  tested  shall  be: 

The  implied  null  hypothesis  is: 

«i^i =02=° 

The  significance  level  shall  be  set  at  0.05. 

‘The  F-test  shall  be  used  to  test  the  differences  between  sample  means.  In  spe- 
cific  terms  the  research  hypothesis  is  that  the  mean  responses  in  the  manual 
mode  will  be  different  than  the  mean  responses  in  the  automated  mode  and  that 
such  a difference  is  not  due  to  sampling  error. 


d.  Enroute  Control . 


The  following  calculations  shall  be  made  for  each  mode.  Max- Automated,  Semi- 
Automated,  and  Alin- Automated  for  both  Air-to-Air  and  Air-to- Restricted  Zone 
conflicts. 


Means 

SD 

C.  P.D. 

Conflict  alert  acknowledge  time 

XX 

XX 

XX 

Conflict  resolution  time 

XX 

XX 

XX 



A total  mean  shall  be  calculated  in  each  mode.  The  hypothesis  to  be  tested  shall 
be: 


^2*  ^3  not  a11  zero 
The  implied  null  hypothesis  is: 

Hl:  = = A=  °- 

The  significance  level  shall  be  set  at  0,05. 

The  F-test  shall  be  used  to  test  the  differences  between  sample  means.  In  spe- 
cific terms  the  research  hypothesis  is  that  the  subject  mean  responses  will  be 
different  across  each  mode  and  that  such  a diffeience  is  not  due  to  sampling 

error. 


e . Handover . 

Handovers  will  occur  in  the  Approach,  Departure*  and  Enroute  functions.  The 
event  (handover /handin)  will  be  controlled  by  the  scenario.  The  following 
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calculations  shall  be  made  for  handovers  in  each  function,  for  min-auto  and 
max-automated  modes. 


Means 

SD 

C.  P.D. 

Handover  alert  acknowledge  time 

XX 

XX 

XX 

Handover  coordination  time 

XX 

XX 

XX 

The  F-test  shall  be  applied  for  testing  a difference  between  means  in  each  of  the 
three  functions.  The  hypothesis  to  be  tested  shall  be: 

% * 0 

The  implied  null  hypothesis  is: 

Hl!^l=/J2  = 0 

The  significance  level  shall  be  set  at  0, 05,  In  specific  terms  the  research 
hypothesis  is  that  the  mean  responses  in  the  min-automated  mode  will  be  dif- 
ferent than  the  mean  responses  in  the  max-automated  mode  and  that  such  a dif- 
ference is  not  due  to  sampling  error. 

11 . RESULTS  COMPARISON 

The  controller  and  system  performance  distributions  obtained  from  the  tests  will 
be  used  to  validate  the  Queuing  Model.  Validation  of  specific  functions  will  be 
performed  by  comparison  with  the  results  of  previous  studies  or  (when  there  are 
no  previous  results  for  a function)  by  comparison  with  predicted  results. 

The  previous  studies  to  be  considered  are: 

• Final  Report  on  the  Evaluation  of  a Semiautomatic  Flight  Operations  Center 

• Army  Tactical  Air  Space  Regulation  System  (ATARS) 

• Design  Plan  Supplement  for  Basic  Semiautomatic  Flight  Operations  Center 

• MTACCS 

• FAA  Controller  task  time  data 

The  methods  of  implementing  air  traffic  control  functions  and  the  scenarios  used 
vary  between  the  above  studies  and  the  ATMAC.  It  is  necessary,  therefore,*  to 
determine  which  study  corresponds  best  for  a particular  function,  and  to  make 
allowance  for  these  differences  in  making  the  comparisons  with  the  data  collected 
on  the  I/A  SIM. 

For  those  ATMAC  functions  where  there  are  no  previous  data  (or  the  degree  of 
correspondence  is  not  acceptable),  the  test  results  will  be  compared  with  pre- 
dicted results.  The  predicted  results  will  be  based  on  analysis  of  the  modeling 
techniques  used  to  generate  the  simulation  and  the  test  procedures  developed  for 
the  test  subjects.  The  analysis  will  separate  the  data  processing  actions  from 
the  test  subject  actions  and  assign  elapsed  time  periods  to  each.  The  data 


363 


processing  times  will  be  derived  from  the  same  or  similar  processing  techniques 
used  in  ATMAC  which  have  correspondence  with  one  of  the  previous  studies.  The 
test  subject  times  will  be  obtained  with  real  time  observations  of  the  test  subject 
performing  the  ATMAC  functions. 

To  validate  the  data,  controller  response  distributions  from  the  previous  studies 
or  from  the  predicted  results  analysis  will  be  statistically  compared  with  the 
corresponding  distributions  from  die  Interactive  simulator.  The  data  will  be 
considered  validated  if  the  comparison  shows  that  both  distributions  are  within 
the- confidence  interval  of  the  selected  population  parameter.  If  the  comparison 
Is  false,  further  attempts  will  be  made  to  justify  the  differences  by  analysis.  In 
the  event  that  adequate  explanations  for  the  differences  are  found,  the  data  ob- 
tained from  the  1/A  SIM  will  be  used  as  input  parameters  to  the  Queuing  Model. 

If,  on  the  other  hand,  inadequate  reasons  for  the  discrepancies  are  found,  the 
l/A  SIM  data  will  be  adjusted  until  the  remaining  differences  can  be  justified. 


GLOSSARY  OF  SYMBOLS 


Ijk 

l« 


at. 


0. 


ij 

:ijk 


H, 


H. 


H, 


2 

a 

ij 

ft 


ai 


ij 


ss, 


- k th  time  response  of  operator  i in  mode  j 

= mean  response  time  for  operator  i in  mode  j (cell  ij  mean) 
= general  mean 

- main  effect  of  operator  i 
= main  effect  of  mode  j 

- Interaction  of  operator  i with  mode  j 

2 

= sample  error,  N(0,  a ) 

= hypothesis:  for  all  i,j 

“ hypothesis:  0^-0  for  all  j 

- hypothesis:  a^-0  for  all  i 

= significance  level  for  the  F-test 

* sample  mean  response  time  for  operator  i in  mode  j 
= general  sample  mean 

= sample  main  effect  of  operator  i 

* sample  main  effect  of  mode  j 

= sample  interaction  between  operator  i and  mode  j 
“ sum  of  squares  for  main  effects  of  the  operators  (o^) 
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SSB 

ssc 

ssE 

*J1 

en 

A 

9n 

N 


= sum  of  squares  for  main  effects  of  the  modes  (/?j) 

= sum  of  squares  for  the  interactions  (y.j) 

= sum  of  squares  for  the  errors 

“ contrast  of  main  effects  of  the  operating  modes  A^) 
= least  square  estimate  of  4^  from  the  sampled  values  of 
■ contrast  of  the  main  effects  of  the  operators 
= least  square  estimate  of  0^ 

= no.  of  observations  from  the  ij-cell 


n — no,  of  operators 

m = no.  of  modes  of  operation 

Means  “ sample  means 
SS  = sum  of  squares 

Q = general  hypothesis  for  the  two-way  classification  analysis  of  variance 

CPD  = cumulative  probability  distribution 
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12. 


nATA  COLLECTION  FROM  THE  INTERACTIVE  SIMULATOR 


Time  measurements  of  controller  interaction  with  simulated  situations  are 
collected  from  the  mtei'active  Simulator  and  used  as  part  of  the  validation  data 
for  the  ATMAC  Queuing  Model.  The  collected  data,  as  applied  to  the  ATMAC 
Queuing  Model,  are  in  the  form  of  distributions  resulting  from  several  trials 
of  each  simulated  situation.  Section  3 describes  the  data  to  be  collected  and 
its  output  format. 

13.  DEVELOPMENT  OF  OTHER  MODEL  VALIDATION  DATA 

V " " ' " 

The  model  validation  data  obtained  from  the  Interactive  Simulator  represents 
only  a portion  of  the  data  which  is  required  to  assure  a totally  valid  queueing 
model.  Such  additional  validation  data  include,  but  are  not  limited  to*  addi- 
tional controller  response  time  statistics  derived  from  other  sources,  usage 
ratet  of  various  classes  of  subsystems,  and  flight  durations  and  durations  of 
discrete  events  as  a function  of  the  scenarios.  Descriptions  of  these  other  vali- 
dation techniques  are  included  in  Section  14. 


14.  ATMAC  QUEUING  MODEL  VALIDATION 

The  model  validation  process  includes  the  following; 

• Use  of  valid  scenarios  by  using  Government  furnished  and  acceptable 
documents. 

• Validation  of  the  ATMAC  Model  structure. 

• Validation  of  the  results  of  certain  simulation  runs  using  data  obtained  from 
the  Interactive  Simulator. 

• Validation  of  simulation  measures  using  empirical  data. 

• Indirect  validation  of  other  data  which  have  not  been  previously  developed. 
The  plan  for  Queuing  Model  validation  is  described  in  this  section. 
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THE  S-METHOD  (SCHEFFE'S  METHOD) 


Whenever  either  of  the  hypotheses  Hi  or  H2  is  rejected  under  the  hypothesis  Q 
(that  is.  Ho  is  rejected),  the  S-method  wiH  be  used  to  determine  which  factors 
contributed  to  the  rejection  and  to  estimate  the  size  of  the  differences  between 
the  parameters.  In  particular,  if  Hi  is  rejected,  then  the  functions 

hr  h-*yl  st<isn> 


will  be  tested  for  significant  differences  from  0 by  use  of  the  least  square  esti- 
mate ^ of  Note  that 


n _ 

- Bf 

£j 


Br 


The  basis  of  the  S-method  is  the  following  fact:  the  probability  is  1-a  that 


where 


and  S is  determined  from  an  F-distribution  with  parameters  (m-1)  and^-mn) 
such  that 


The  statement  above  implies  that  4*ij  is  significantly  different  from  0 if 

A _ A / A \ 
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If  H2  is  rejected  under  the  hypothesis  Q,  then  the  functions 
*1f  = Qi“ar  Hi^sn* 

will  be  tested  for  significant  differences  from  0.  The  procedure  is  similar  to 
that  outlined  above  with 


en  Ai~Ae 


and 


SSE 

" N.-nm 


„ m * - m - 

1 _1 1_  _ 1 

2 N,.  2 N* , 

m j=i  ij  m j=i  li 


\ • 
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Whenever  either  of  the  hypotheses  Hi  or  H2  is  rejected,  Tukey's  Method  (here- 
after, simply  the  T -method)  will  be  used  to  determine  which  factors  contributed 
to  the  rejection  and  to  estimate  the  size  of  the  differences,  m particular,  if  Hi 
is  rejected,  then  the  functions 

= < { < m 

will  bo  tested  for  significant  differences  from  0 by  use  of  the  least  square  esti- 
mate ^ £ of  4^.  Note  that 

=/5i 

Let  a satisfy  0 < a < 1.  The  T-method  is  based  upon  the  following  theorem  (due 
to  Tukey):  The  probability  is  1 -.<*  that 

| t ” ^ j t 5 TS  5 J < * £ TO* 

where 


and  T is  chosen  to  satisfy 

P (t  < b T)‘-  1 - of, 
b2  = n ^Uj'1  + Uf1} 

where  t has  a student  t-distribution  with  parameters  m and  (Nt-mn).  Thus, 
there  are  significant  differences  In  then’s  if 

A 

0 < 4^  - TS 

for  some  j and  1 . 
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If  the  hypothesis  H»  is  rejected,  then  the  functions 
9^  = 0^ -Off,  1 < i < 1 s n 

wiU  be  tested  for  significant  differences  from  0.  The  procedure  is  the  same  as 
that  outlined  above  except  that  T is  chosen  to  satisfy 

p (t  s a T)  = 1-of, 

a^rn^Wf-1) 

where  t has  parameters  n and  (N^  - inn). 
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15.  -INTERACTIVE  SIMULATOR  DATA  ANALYSIS 

The  following  paragraphs  contain  a description  and  summary  of  the  results 
from  the  statistical  analysis  of  the  Interactive  Simulator  data  proposed  in  the  Valida- 
tion Test  Plan  (VTP).  As  stated  in  the  VTP,  each  of  the  ATMAC  functions  simulated 
on  the  Interactive  Simulator  generates  data  which  represent  controller  response 
times,  switch  actions  and  efficiency  of  operation.  The  goal  of  the  statistical  analysis 
of  this  data  is  to  gather  meaningful  data  concerning  the  human  response  times  which 
can  be  applied  to  the  ATMAC  Queueing  Model. 

a . Description  of  the  Data. 

The  primary  method  of  the  statistical  analysis  of  the  I/A  data  was  the  two-way 
analysis  of  variance  scheme  outlined  in  the  VTP;  also  see  Henry  Scheffe':  The  Anal- 
ysis of  Variance  (New  York;  John  Wiley  & Sons,  Inp. ; 1959).  The  analysis  of  variance 
scheme  identifies  simultaneously  differences  in  operator  performance  between  the 
individual  operators  and  between  levels  of  automation.  The  data  for  those  operators 
which  consistently  displayed  response  times  significantly  different  from  the  majority 
was  deleted  from  the  data  set;  the  analysis  then  was  repeated  for  the  remaining  data. 

In  this  way  it  was  possible  to  obtain  information  concerning  the  distribution  of  opera- 
tor response  times  (the  mean,  standard  deviation  and  cumulative  distribution  func- 
tion) in  each  level  of  automation  and  to  identify  the  differences  in  the  response  times 
which  were  a result  of  mode  manipulation  rather  than  sampling  errors, 

b . Interactive  Simulator  Data. 

The  data  from  the  Interactive  Simulator  in  the  max-automated  automation 
level  consists  of  an  average  response  time  together  with  the  number  of  samples, 
standard  deviation,  and  cumulative  distribution  function  for  each  operator  with  respect 
to  each  task. 

In  order  for  the  results  of  the  two  factor  analysis  to  be  meaningful,  it  is  nec- 
essary that  the  response  times  from  the  different  levels  of  automation  represent 
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performance  of  the  same  (or' at  least  similar)  tasks.  Unfortunately,  in  many  cases 
the  data  from  the  interactive  simulator  in  the  max-auto  mode  do  not  correspond 
directly  with  the  data  from  the  interactive  simulator  in  the  min'-auto  mode  of  opera- 
tion. Consequently  it  was  necessary  in  some  cases  to  group  or  identify  two  separate 
tasks  as  one  task;  in  other  cases  it  was  necessary  to  add  the  response  times.  The 
formulae  for  these  manipulations  are  given  in  the  following  paragraph. 

As  in  the  VTP,  let  the  operators  be  indexed  by  the  subscript  i and  the  level  of 
automation  by  the  subscript  j.  For  a given  task,  represents  the  number  of  times 
operator  i performed  the  task  in  level  ji  and  cr^  represent  the  corresponding  mean 
and  standard  deviation  of  the  response  times.  Whenever  it  was  necessary  to  group  or 

identify  two  tasks,  the  new  statistics  N..,  A..,  tr.  were  computed  from  the  original 

U U 

statistics  N^»  X^\  and  x(^)  follows: 

ij  U ij  ij  ij  ij 


* 


a(1)+n<2)  \ (2) 

^ j ij  ij  Nij  *ij 


* 


Whenever  it  was  necessary  to  add  response  times,  the  new  statistics  were  computed 
according  to: 


= minimum 


K’.  »?’]• 


A = X<X>  + X<2> 
Aij  ij  ij  * 


In  addition  to  the  discrepancy  between  the  min~auto  and  max-auto  modes  with 
respect  to  the  definition  of  the  tasks,  there  is  another  fundamental  difference  in  the 
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data.  A response  time  in  the  max-auto  mode  includes  the  time  necessary  to  perform 
two  distinct  phases  of  the  task;  namely  a decision-making  process  in  which  the  appro- 
priate action  is  determined  followed  by  the  physical  or  mechanical  processes  (i,e. , 
switch  actions)  necessary  to  implement  the  decision.  In  contrast  to  the  max-auto 
mode,  the  min-auto  mode  response  times  do  not  include  the  times  for  the  decision 
making  process  because  only  times  for  voice  communications  are  available.  In  order 
to  compensate  for  this  difference  between  the  data  from  the  two  modes,  the  manual 
mode  response  times  were  increased  by  25  percent.  No  attempt  was  made  either  to 
justify  the  magnitude  of  the  increase  or  to  investigate  the  sensitivity  of  the  results  to 

the  magnitude  of  the  increase. 

c . Analysis  of  Variance  : Data . 

The  two- factor  analysis  of  variance  scheme  was  used  to  analyze  the  Approach 
Control,  Departure  Control,  Enroute  Control  and  Handover  data.  Valid  Flight  Plan 
data  was  collected  from  two  operators  only;  consequently  the  means  and  standard 
deviations  for  the  response  times  were  computed  with  no  additional  data  evaluation. 

When  the  analysis  of  variance  methods  were  used  to  analyze  the  data  for  a 
particular  ATMAC  function,  a computer  program  (a  listing  is  included  with  the  data) 
was  executed  to  perform  the  necessary  computations.  Each  time  the  program  was 
executed,  the  data  described  in  the  following  sections  was  printed. 

(1)  For  each  operator,  the  mean  response  time  the  number  of  observations 
N^,  the  standard  error  of  the  mean  and  a 90  percent  confidence  interval 
for  the  true  mean  /a-  of  the  distribution  is  printed  for  each  method  of 
operation.  The  standard  error  of  mean  is 


The  confidence  interval  is  computed  by  considering 
Pr  (-  b < T < b)  « 0. 90 
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where 


has  a student’s  t - distribution  with  parameter  - 1.  The  variable 
represents  the  true  mean.  The  probability  is  0. 90  that 

VbB«<#£u<;VbV 

(2)  For  each  mode  of  operation  under  consideration,  the  mean  operator 
response  time,  the  standard  error  of  the  mean,  and  a 90  percent  confi- 
dence interval  for  the  true  mode  mean  are  printed.  The  first  value  of  the 
mode  mean  A.  is  computed  according  to 


n 


A -~ 
Ai  n 


4m*  A {4 


1=1  ^ 

where  n is  the  number  of  operators  considered.  The  second  value  of  the 
mode  mean  A^  is  given  by 


n 


V 

where 


= -!£  N X 

j n.  Kij  xij 


n 


N = £ N. 
J i=l 


Similarly,  two  values  of  the  standard  error,  and  s^,  are  given; 


n 


82 L_  , 

i (n-X)2  k l>' 

if— SEnJ. 

* (N.-l)  1=1  1)L 


X2  +«-2'1  " 
ij  +<rU 


]-sr 
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Two  confidence  intervals  for  the  true  mode  mean/*,  are  computed  by 

J 

considering  the  student  t - statistics 


t-V^I 

8) 


which  have  parameters  n-1  and  N^-l,  respectively. 

( 3 ) The  analysis  of  variance  parameters  are  printed  also.  These  include  the 
general  mean/i,  the  variations  in  the  mean  due  to  the  operators  (A -main 
effects),  the  variations  /?.  in  the  mean  due  to  the  automation  levels 
(B-main  effects),  and  the  interactions  Y^.  Recall  that  the  underlying 
assumption  of  the  analysis  of  variance  scheme  is  that  the  individual 
response  times  satisfy 

xijk=  #t+“i+VYij +c> 

Where  € is  a random  sample  from  a normal  distribution  with  mean  O and 
2 

variance  <r  . In  addition  to  these  parameters,  five  F-statistics  are 
printed.  (See  Table  v ) . 

16,  SUMMARY  OF  RESULTS 

In  the  following  sections,  the  mean  response  times  together  with  the  corre- 
sponding standard  deviations  are  given  for  the  various  ATMAC  functions.  Except  for 


the  Rlight  Plan  data,  the  analysis  of  variance  scheme  was  used  to  isolate 
those  operators  who  consistently  displayed  response  times  significantly  different  (as 
determined  by  the  S or  T method  of  multiple  comparison).  The  mean  and  standard 
deviation  of  the  response  times  were  computed  by  deleting  the  data  of  the  operators 
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Table  V.  F-STAT1STICS 


Statistic 

Degrees  of  Freedom 

Use 

F(D 

mn  - n - m + 1,  Nt  - mn 

Test  Hq  against  Q 
(Additivity:  7^  ~ 0) 

F(2) 

n - 1,  Nt  - m » n + 1 

Test  H„  against  n 
= 0) 

F(3) 

m-1,  ~ m - n + 1 

Test  against  £2  HQ 

F(4) 

n — 1,  Nj.  - mn 

Test  H_  against  Q 
= 0) 

F(5) 

m - 1,  Nt  - mn 

Test  against  Q 
Oj  = 0) 

m » number  of  automation  levels 
n - number  of  operators 

= total  number  of  observations 

mentioned  above.  A computer  print-out  of  the  original  data  sets  as  well  as  the  final 
data  sets  is  included. 

As  was  stated  previously,  the  Flight  Plan  means  and  standard  deviations  were 
computed  by  using  all  of  the  available  data. 
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17 . FLIGHT  PLAN  ENTRY/MODIFICATION/CLEARANCE 


The  following  is  based  upon  the  data  collected  from  operators  Olson  and 

Wilson. 


Table  VI.  FLIGHT  PLAN  DATA 


Function 

Mean 

Std.  Dev. 
of  Population 

Std.  Dev. 
of  Mean 

No.  of  Observations 

Entry  Time 
(Seconds) 

209.50 

62.58 

11.43 

30 

Modification 

Time 

52.05 

21.75 

4.44 

24 

Clear-Reject 

Time 

130. 12 

67.19 

14.66 

21 

18.  APPROACH  CONTROL 

The  analysis  of  variance  program  was  used  to  compare  the  max -automated 
mode  of  operation  with  the  min-automated  mode  of  operation  for  two  data  sets.  In  the 
first  data  set,  which  is  labeled  "Accept  Hand-ln",  the  automatic  mode  data  consists 
of  the  I/A  SIM  data  bearing  the  same  label;  the  min-auto  mode  data  consists  of  the 
sum  of  the  following  tasks: 

• Establish  contact  with  adjacent  sector  controller 

• Complete  handover 

• Pilot  establishes  contact 

The  second  data  set,  which  is  labeled  "Approve  Vector  Instructions",  consists  of  the 
I/A  SIM  data  with  the  same  name  for  the  max -auto  mode  together  with  the  following 
grouped  min-auto  mode  data: 

• Provide  vector  instructions 

• Issue  formation  breakup  command. 
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In  both  cases,  the  analysis  showed  that  the  data  did  not  satisfy  the 
criterion  for  additivity;  that  is,  Hq  was  rejected.  Again,  in  both  cases, 
all  but  two  operators  were  eliminated  from  consideration;  for  both  cases, 
the  final  mean  response  time  in  the  max-auto  mode  was  significantly  (by 
the  S-method)  less  than  the  mean  response  time  in  the  min-auto  mode. 


Table  VII.  APPROACH  CONTROL  DATA:  MAX-AUTOMATED  MODE 


Function 

Mean 

Std.  Dev. 
of  Population 

Std.  Dev. 
of  Mean 

No.  of 

Observations 

Accept  Hand-In 

7.34  sec. 

5.15 

0,72 

51 

Approve  Vector 
Instructions 

3.38  sec. 

4.45 

0.58 

59 

Resolution  of 
Separation 
Alert 

42.33  sec. 

29.85 

8.62 

12 

Minimum  Approach 
Separation 
Distance 

553.57  meters 

310.83 

89,73 

12 

Table  VIII.  APPROACH  CONTROL  DATA:  MIN-AUTOMATED  MODE 


Function 

Mean 

(sec.) 

Std.  I 
of  Pop. 

)ev 

of  Mean 

No.  of  Observations 

Provide  Vector 
Instructions 

5.67 

2.53 

■ 

0.15 

294 

Formation  Backup 
Command 

4.72 

4.21 

0.84 

25 

Establish  Contact 
with  ad j . sector 

1.55 

1.71 

0.41 

17 

Complete  Handover 

10,94 

5.10 

1,20 

18 

Pilot  establishes 
contact 

3.76 

1.44 

0.29 

24 
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19/  DEPARTURE  CONTROL. 


The  analysis  of  variance  program  was  used  to  compare  the  max-auto  mode  of 
operation  with  the  min-auto  mode  of  operation  for  two  sets  of  data.  In  the  first  data 
set,  which  is  labeled  "Alert  Acknowledge",  the  max-auto  mode  data  consists  of  the 
I/A  SIM  data  with  the  same  name;  the  min-auto  mode  data  consists  of  the  sum  of  the 
following  steps; 

• Receive  call  from  tower 

• Request  and  receive  departure  data 

In  the  second  data  set,  which  is  labeled  "Approve  Vector  Instructions",  the  max-auto 
mode  data  consists  of  the  I/A  SIM  data  with  the  same  name;  the  min-auto  mode  data 
consists  of: 

• Provide  Vector  Instructions 

• Provide  Departure  Instructions 

For  the  "Alert  Acknowledge"  data,  the  hypothesis  of  additivity,  that  is  H^, 
was  accepted.  The  mean  response  time  for  the  max-auto  mode  was  significantly  less 
than  that  of  the  min-auto  mode  (as  determined  by  the  S or  T method).  In  the  second 
data  set,  the  hypothesis  HQ  was  rejected;  however  the  max -auto  mode  response  time 
also  was  found  to  be  significantly  less  than  the  min-auto  mode  response  time. 

20.  ENROUTE  CONTROL 

The  analysis  of  variance  program  was  used  to  compare  the  operator  response 
times  for  conflict  resolution  in  the  max -automated,  semi-automated,  and  min- 
automated  modes  of  operation.  The  max-auto  mode  data  consists  of  the  I/A  SIM  data 
for  the  resolution  time  of  air-to-air  conflicts  and  air-to-restricted  zone  conflicts. 

The  semi-automated  mode  data  consists  of  all  data  for;  "Provide  vector  Instructions", 
The  min-auto  mode  data  is  similar  to  that  of  the  semi -auto  mode. 
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Table  IX.  DEPARTURE  CONTROL  DATA:  MAX- AUTOMATED  MODE 


Std. 

Dev. 

Function 

Mean 

of  Pop. 

of  Mean 

No.  of  Observation 

Approve  Vector 
Instructions 

2. 80  sec. 

D 

0.28 

122 

Departure  Alert 
Acknowledge 

4. 96  sec. 

4.18 

0.47 

79 

Resolution  of 
Separation  Alert 

41.52  sec. 

26.03 

8.23 

10 

Minimum  Departure 
Separation 
Distance 

362.44  meters 

440.92 

139.43 

10 

Table  X.  DEPARTURE  CONTROL  DATA:  MIN- AUTOMATED  MODE 


function 

Mean 

Std. 

Dev. 

(sec. ) 

of  Pop. 

No.  of  Observations 

Provide  Vector 
Instructions 

5.63 

2.65 

0.16 

262 

Redelve  Call  from 
Tower 

1.81 

1.43 

0.19 

55 

Request  and  Receive 
Departure  Data 

6.49 

2.81 

0.38 

56 

Provide  Departure 
Instructions 

6.56 

4.79 

0.56 

74 

Successive  application  of  the  analysis  of  variance  scheme  eliminated  all  but 
three  operators.  On  the  basis  of  the  data  for  these  three,  the  hypotheses  and 
were  rejected.  By  the  S-metbod,  the  response  times  in  the  semi-automated  mode 
were  found  to  be  significantly  less  than  those  of  the  other  two  modes;  however  there 
was  no  significant  difference  between  the  max-auto  and  min-auto  mode  response 
times. 

In  addition  to  the  conflict  resolution  data,  the  analysis  of  variance  program 

was  exercised  to  compare  the  operator  response  times  for  the  max-auto  mode  against 
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the  response  times  for  the  semi-auto  mode  for  the  Air-to-Air  and  the  Air~to-Restricted 
Zone  Conflict  Alert  Acknowledge  Time  data.  Both  data  sets  were  taken  directly  from 
the  X/A  SIM  output  data. 

The  analysis  of  variance  indicates  that,  in  both  data  sets,  is  accepted  and 
is  rejected.  Relative  to  the  S-method,  the  mean  response  time  in  the  max-auto 
mode  is  significantly  less  than  that  of  the  semi-automated  mode. 


Table  XI.  ENROUTE  CONTROL  DATA:  MAX-AUTOMATED  AND  SEMI -AUTOMATED 

MODES  (DATA  COLLECTED  BY  COMPUTER  PROGRAM) 


Mean 

Std, 

Dev. 

[ 

! 

Function 

(sec. ) 

of  Pop, 

of  Mean 

No,  of  Observations 

Conflict  Alert  Ack. 
Air-Air  (Max-Auto) 

15.31 

8.93 

0.49 

332 

(Semi-Auto) 

19.15 

9.56 

0.52 

339 

Air-Restricted  Zone 
(Max -Auto) 

14.94 

9.04 

1.  05 

74 

(Semi-Auto) 

19. 11 

10.  07 

1. 28 

62 

Conflict  Resolution 
Air-Air 

7.69 

4.20 

0.26 

271 

Air-Restricted  Zone 

7.20 

3.00 

0.  95 

10 

Table  XII.  ENROUTE  CONTROL  DATA;  MIN-AUTOMATED  AND  SEMI-AUTOMATED 
MODES  (DATA  COLLECTED  FROM  VOICE  TAPES) 


Mean 

Std, 

, Dev. 

Function 

(sec. ) 

of  Pop. 

of  Mean 

No.  of  Observations 

Provide  Vector 
Instructions 
(Min-Auto) 

6.69 

3.37 

0. 26 

172 

Provide  Vector 
Instructions 
(Semi-Auto.  Mode) 

5.30 

2.55 

0. 17 

225 

Controller  calls  adjacent 
facility  and  completes 
handover  (Semi -Auto) 

3.07 

MM 

0.  09 

111 
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21.  HANDOVER 

For  each  of  the  three  general  functions  - Approach,  Departure  and  Enroute 
Control  — data  was  collected  for  the  associated  handover  functions.  The  analysis  of 
variance  program  was  exercised  in  order  to  compare  the  max-automated  with  the 
min-automated  mode  response  times  for  the  Approach  and  Departure  Control  functions; 
the  program  was  used  to  compare  the  response  times  for  all  three  levels  of  automa- 
tion for  the  Enroute  Control  function.  In  all  of  the  three  data  sets  bearing  the  label 
"Handover",  the  max-auto  data  (and  the  semi-auto  mode  data  for  the  Enroute  Control 
function)  consists  of  the  sum  of  the  response  times  labeled  "Handover  Alert  Acknowl- 
edge" and  "Handover  Coordination  Time".  In  the  data  set  for  the  Approach  Control 
function,  the  min-auto  mode  data  is  the  sum  of  the  following  Steps: 

• Establish  contact  with  adjacent  sector  controller 

• Complete  Handover 

• Pilot  establishes  contact 
together  with  the  sum  of: 

• Initiate  tower  handover 

• Complete  tower  handover. 

For  the  Departure  Control  function,  the  min-auto  mode  data  is  the  grouped  data  for: 

• Pilot  establishes  contact 

• Contact  adjacent  sector  controller 

• Complete  handover. 

Finally,  for  the  Enroute  Control  function,  the  min-auto  mode  data  consists  of  the  sum 
of  the  data  for.* 

• Establish  contact  with  adjacent  cector  controller 

• Complete  handover 

• Pilot  establishes  contact 


382 


together  with  the  sum  of  the  data  for: 

• Contact  adjacent  sector  controller 

• Complete  handover. 

For  both  the  Approach  Control  function  and  the  Departure  Control  function, 
the  hypothesis  HQ  was  accepted  and  the  hypothesis  was  rejected.  In  both  cases, 
the  S-method  indicated  that  the  mean  response  time  for  the  max -auto  mode  is  signifi- 
cantly less  than  the  mean  response  time  for  the  min-auto  mode.  On  the  other  hand, 
the  analysis  of  variance  of  the  Enroute  Handover  data  resulted  in  the  rejection  of 
and  the  acceptance  H^,  that  is,  there  is  no  significant  difference  in  the  response  times 
for  the  three  modes  of  operation  when  tested  by  the  S-method. 


table  XIII.  HANDOVER:  MAX-AUTOMATED  AND  SEMI-AUTOMATED  MODES 


Mean 

Std. 

Dev. 

No. 

Mode 

Function 

(sec. ) 

of  Pop. 

of  Mean 

of  Observation 

Approach  Control 

1 

6.63 

7.20 

0.45 

259 

(Max -Automated) 

2 

7.91 

7.20 

0.44 

268 

Departure  Control 

1 

6.35 

4.52 

0.39 

135 

(Max-Auto) 

2 

7. 15 

4.54 

0.39 

135 

Enroute  Control 

1 

9.25 

10.80 

0,  71 

234 

(Max-Auto) 

2 

9.33 

7.83 

0.51 

234 

(Semi-Auto) 

1 

9.50 

11. 89 

■SI 

228 

2 

9.54 

9.52 

0. 63 

228 

Function  1 — Handover  Alert  Acknowledge 
Function  2 - Handover  Coordination  Time 
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Table  XIV.  HANDOVER:  MIN-AUTOMATED  MODE 


Function 

Mean 

Std,  Dev. 

No,  of 

Observation 

of  Pop, 

of  Mean 

Approach  Control 

Initiate  Tower  Handover 

1. 71 

2.77 

0.42 

43 

Complete  Tower  Handover 

9.77 

3.89 

0.62 

39 

Departure  Control 

Pilot  establishes  contact 

4.05 

1.86 

0. 19 

101 

Contact  adjacent  sector  controller 

3,64 

2.87 

0.29 

100 

Complete  Handover 

5.73 

3.82 

0.38 

100 

Enroute  Control 

Establish  contact  with  adjacent  controller 

2.44 

2.28 

0.23 

96 

Complete  handover 

9,  95 

5,16 

0.53 

95 

Contact  adjacent  sector  controller 

1.96 

0.75 

0.10 

62 

Complete  handover 

; n.24 

5.42 

0.69 

61 

Pilot  establishes  contact 

4.19 

2,11 

0. 22 

89 
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22.  DETAILED  ANALYSIS  OF  THE  DATA  FOR  APPROACH  CONTROL:  ACCEPT  HAND-IN 


The  original  data  set  for  the  Approach  Control:  Accept  Hand-in 

function  is  contained  in  the  Appendix  (Al.l).  The  results  of  the  applica- 
tion of  the  analysis  of  variance  program  are  given  in  the  Appendix  (A1.2). 

A brief  inspection  of  the  F-statistics  in  the  Appendix  (A1.2)  will  indicate 
that  the  hypothesis  Hz  will  be  rejected  by  the  F-test.  Table  XV  contains 
the  values  of  the  contrasts 

hi  m ai  - al 

A A 

for  \(s  i < 1 £ 7;  the  value  of  cr  (0^)  appears  immediately  below  0^. 

According  to  Scheffe's  method,  6-q  will  be  significantly  different  from 

0 if 

(1)  |011|  > S a (eu) 

where  S is  defined  by 

(2)  Pr  (F  £ S2/(n  - 1).  - 1 - a - .95 

for  an  F-distribution  with  (n  - 1)  and  (Nt  - ran)  degrees  of  freedom.  In 
this  case  S *3.55.  From  Table  XV  it  is  clear  that  the  inequality  (1)  is 
valid  for  1*4,  i - 1,  2,  3,  5,  6,  7. 

The  Appendix  (A1.3  and  A1.4)  contains  the  results  of  the  analysis  of 
variance  with  the  data  for  operator  4 deleted.  The  value  of  S in  equation 
(2)  is  3.33;  from  Table  XVI  it  is  clear  that  inequality  (1)  is  valid  for 
1-4,  i-1,  2,  3. 

The  Appendix  (A1.5  and  A1.6)  contains  the  results  of  the  analysis  of 
variance  with  the  data  for  operators  4 and  5 deleted.  Since  the  value  of 
F(l)  is  7.12,  the  hypothesis  H0  (Additivity)  is  rejected;  thus  F(4)  is  used 
to  test  Hz.  The  degrees  of  freedom  for  F(4)  are  4 and  128.  Now 

Pr  (F  > f ) - o * .05 

Implies  that  f is  approximately  2.45.  Since  the  value  of  F(4)  is  2.84, 

Hz  is  rejected  again.  The  value  of  S in  (2)  is  3.10.  Table  XVII  shows 
that 

l«ijl  5 So  (®ij) 

for  all  i,  j.  Consequently,  the  rejection  of  the  hypothesis  H_  Is  due  to 
a linear  combination  of  the  a^'s  other  than  a simple  contrast  (that  is, 
the  Gjj* s) . However,  the  value  of  the  sample  standard  deviation  ojj  .for 
operator  4 (6  in  the  Appendix  (Al.l))  is  21.4;  the  values  of  Ojj  for  the 
other  operators  in  the  Appendix  (A1.5)  are  3.8,  4.9,  5.1,  and  6.5,  re- 
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spectively.  Therefore,  the  data  for  Operator  4 was  deleted  from  the  data 
set  and  the  analysis  was  repeated. 

The  results  of  the  analysis  are  given  in  the  Appendix  (Al.7  and  A1.8). 

The  hypothesis  H was  rejected  again.  The  value  of  S in  (2)  is  2.87;  the  in- 
equality (1)  holds  for  6^4  and  634.  The  data  for  operator  4 (7  of  the  original) 
was  deleted  and  the  analysis  repeated.  The  results  are  given  in  the  Appen- 
dix (A1.9  and  A1.10).  The  appropriate  value  of  S in  (2)  is  2.49;  from 
Table  XIX  it  is  clear  that  (1)  holds  for  612*  The  data  for  operator  2 was 
deleted. 

The  analysis  of  variance  program  was  applied  to  the  data  for  the  re- 
maining two  operators,  the  results  are  given  in  the  Appendix  (Al.ll  and 
A1.12).  The  F statistic  in  the  Appendix  (A1.12)  indicate  that  H0  and  H2 
will  be  accepted  and  that  will  be  rejected.  The  value  of  ^12  is  - 9.01 

where 


■ &1  - B2. 

The  value^of  S in  (2)  (where  m replaces  n)  is  approximately  1.6.  . Since  the 
value  of  o is  1.12,  B^  is  significantly  larger  than  B2* 
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Table  XV  . MULTIPLE  COMPARISON  BY  THE  S-METHOD 


A 

Line  2 Is  0 (0^) . 
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Table  XVI . MULTIPLE  COMPARISON  BY  THE  S-METHOD 


7.98 

2.24 


10.29 

2,09 


5 

6 

5.11 

2.34 

5.38 

2.17 

1.77 

2.46 

2.04 

2.29 

4.08 

2.33 

4.35 

2.15 

- 6.21 
2 . 38 

- 5.94 
2.21 

• ..  i 

.27 

2.26 

-■Table  XVII.  MULTIPLE  COMPARISON  BY  THE  S-METHOD 


2 

3 

3.33 

1.03 

2.01 

1.87 

- 2.31 

1.19 

5.11 

2.14 

1,77 

2.25 

4.08 

2.13 


5.38 

1.98 

2.04 

2.09 

4.35 

1,96 

.27 

2.22 


Table  XIX . MULTIPLE  COMPARISON 
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90%  CONFIDENCE  IN T f _ 5,32  14,35 

26, 05  24,47 
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. .2, 463_ 

STANDARD  ERROR  = 16,4669 

^STATISTICS  “ 10,0233 
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~ N UMBER  OF ’ mod ES  = 2 

TOTAL  NUMBER  OF  OBSERVATIONS  • 
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A FLEXIBLE,  GENERAL  PURPOSE  COVARIANCE  COMPUTER  PROGRAM 

Clifford  J.  Maloney 
and 

Lucille  A.  Carver 
Food  and  Drug  Administration 
Bethesda,  Maryland  20034 

I.  Introduction : Heraclitus,  five  centuries  before  the  birth 

of  Christ,  maintained  that  no  one  steps  in  the  same  river  twice;  yet 
"enduring  as  a mountain"  is  an  ancient  simile  of  permanence  though 
rivers  are  older  than  the  mountains  they  drain.  Life  is  lived  in  a 
multi-faceted,  ever  changing  milieu;  yet,  if  the  past  is  to  be  our 
guide  to  the  future,  that  can  only  be  because  the  future  is  like  the 
past.  It  is  the  invariant  features,  not  the  ephemeral,  that  matter. 
Constancy  adheres  to  substance,  properties,  and  relations.  Father 
or  son  denotes  no  particular  individual,  but  the  father-son  relation 
is  fixed. 

Strictly  speaking,  substance  is  neither  accessible  to  man,  nor 
needed  in  the  conduct  of  affairs.  The  scientific  revolution  in  es- 
sence was  an  appreciation  of  that  fact.  Substance  is  an  intellectual 
construct  whose  constancy  or  lack  of  it  is  inferred  from  constancy  or 
variation  in  properties  or  relations.  The  latter  two  alone  receive 
the  professional  attention  of  scientists,  though  scientists  have 
varying  appreciation  of  the  validity  of  that  fact. 

In  turn,  properties  and  relations  are  qualitative  or  quantita- 


tive.  Each  is  often  directly  apparent,  and  where  that  is  so,  have 
been  known  since  antiquity.  But  where  the  qualitative  is  inferred 
from  quantitative  observations  it  was,  in  practice,  necessary  to 
await:  (1)  the  discovery  of  which  quantitative  relations  throw 

light  on  which  qualitative  properties  or  relations  and  (2)  the  de- 
velopment of  appropriate  quantitative  scales  and  devices. 

Until  recently  this  latter  process  was  carried  out  informally, 
each  quantitative  construct  being  worked  out  anew  with  no  attention 
to  accumulated  experience  until  Helmholtz  (5)  developed  a general 
theory  of  mensuration.  An  adequate  elementary  treatment  is  now 
available  (2) . 

A complementary  advance  proceeds  in  the  opposite  direction. 

While  it  is  often  necessary  to  discover  which  "instrumental''  variables 
to  observe  in  order  to  measure,  control,  or  understand  a quantity  of 
interest,  it  is  equally  imperative  to  learn  which  plausible  but  irrele- 
vant extraneous  variables  to  ignore.  The  phases  of  the  moon,  the  con- 
junctions of  the  planets,  the  services  of  soothsayers  (except  in  eco- 
nomics) are  easily  recognized  as  such.  But  choice  of  treatment  or 
prognosis  for  a diseased  individual  is  a constant  reminder  that  the 
problem  remains  in  more  difficult  circumstances.  In  essence,  statis- 
tical method  is  the  form  in  which  this  process  is  most  efficiently 
carried  out. 

In  statistical  terms , we  express  constancy  of  a property  by 
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gome  "measure  of  central  tendency";  often  a mean.  And  for  many  pur- 
poses this  is  not  only  "good  enough"  but  the  best  we  can  do.  Thus, 
as  we  seek  to  dimension  a doorway,  we  cannot  vary  its  height  as  each 
person  passes  through,  but  must  adopt  some  one  height  convenient  for 
almost  everyone,  but  seldom  for  all,  unless  monumental  considerations 
determine  the  answer  and  not  the  heights  of  individuals. 

For  clothing  this  doesn’t  work — and  fortunately  another  solution 
is  feasible,  though  far  from  ideal.  Here  a multidimensional  complex 
of  continuously  varying  measures  are  reduced  to  a limited  set  of 
"sizes".  This  same  procedure,  not  always  recognized  as  such,  lies 
back  of  most  if  not  all  applications  of  analysis  of  variance.  By 
treating  one  variate  or  cluster  of  variates  as  a qualitative  cate- 
gorization, a multivariate  problem  is  made  univariate  within  classes. 
Much,  though  not  all,  of  the  advantage  in  doing  so  lies  in  the  re- 
sultant simplification  of  the  arithmetic  required  for  an  effective 
analysis  of  the  resulting  data.  The  advent  of  computers,  where 
available,  greatly  reduces  the  attraction  from  this  point  of  view. 

In  addition,  data  must  frequently  be  analyzed  which  does  not  admit 
the  arithmetic  simplification  of  analysis  of  variance. 

Table  1 sets  out  the  four  situations.  In  line  1 all  factors  are 
held  constant  except  the  variate  of  interest.  This  corresponds  to 
the  "vary  one  factor  at  a time"  prescription  formerly  regarded  as 
the  scientific  method,  and  still  appropriate  in  some  circumstances. 
The  second  applies  wherever  all  but  one  factor  varies  by  jumps  or 


can  be  made  to  do  so— the  analysis  of  variance  situation.  The  contri- 
bution of  American  Experiment  Station  scientists,  in  particular,  J. 

Arthur  Harris,  to  the  development  of  the  concept  of  generalized  variance^ 
and  to  the  now  universally  applied  calculational  algorithm^has  dropped 
from  sight.  Moreover,  the  extension  of  the  number  of  class  means 
above  the  two  treated  by  Student  has  opened  the  Pandora’s  box  of  mul- 
tiple comparisons.  My  own  contribution  to  this  debate  is  given  in  (6). 

But  so  soon  as  regression  coefficients  are  calculated  the  usual 
plethora  of  statistical  questions  arises.  Is  each  significant?  Do 
they  differ?  Is  there  a significant  regression  of  the  group  means? 

How  is  it  related  to  the  within  group  regressions?  And  so  on.  It  is  a 
curious  historical  accident  that  the  usual  textbook  presentation  of  co- 
variance  tend  to  regard  these  questions  as  rarely,  if  ever  of  interest 
and  the  use  of  the  covariance  technique  as  helpful  primarily  (1)  to 
place  group  means  on  a comparable  basis,  and  (2)  to  reduce  the  estimate 
of  residual  error.  This  attitude  is  in  marked  contrast  to  the  insis- 
tence on  the  examination  of  the  parallelism  of  regression  in  the  close- 
ly related,  but  presentationally  and  computational  separate,  technique 
of  bioassay.  The  textbook  proclivity  to  spare  the  novitiate  "unneces- 
sary”  complexity  by  treating  only  "the  most  common"  cases  subjects  him, 
in  the  next  stage,  to  a seemingly  endless  succession  of  unconnected  va- 
riations in  technique  neither  mutually  organized  nor  suggesting  what 
further  developments  are  likely  or  would  exhaust  the  possibilities. 

Yet  to  do  so  only  requires  setting  each  of  the  parameters  in  the  last 
column  of  the  fourth  row  of  Table  1 to  (a)  known  values,  (b)  known 
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relations  or,  (3)  unrestricted  freedom. 

The  following  section  presents  from  first  principled,  and  somewhat 
more  detail  than  is  usual,  those  aspects  of  covariance  provided  for  in 
the  computer  program  which  is  the  principle  topic  of  this  paper. 

XI.  Covariance:  Covariance  analysis  is  a standard  statistical 

technique  widely  explained  in  statistical  texts  (Snedecor  and  Coch- 
ran, (10) » Steel  and  Torrie,  (11)),  though  admitting  elaborations  for 
special  circumstances  for  which  reference  must  be  made  to  the  journal 
literature.  An  experiment  is  never,  or  almost  never,  conducted  in 
complete  ignorance  of  every  aspect  of  the  behavior  of  the  material 
under  study.  The  use  of  linear  covariance  does  not  assume  that  the 
relation  between  the  dependent  and  independent  variate  is  strictly 
linear,  but  only  that  it  is  good  enough  for  the  purpose  at  hand.  Use 
of  a higher  order  polynomial  or  a monotone  function  of  the  readings 
differs  only  In  computational  complexity.  It  is  neglected  here  in  the 
interest  of  simplicity  while  covering  the  most  common  case.  There  is 
also  an  implied  assumption  that  accidental  unobserved  factors  did  not 
disturb  the  progress  of  the  study  (i.e.,  that  departures  from  the 
assumed  linear  relation  are  homogeneous  in  the  probability  sense) . 

That  such  departures  from  the  linear  relation  are  normally  distributed 
(perhaps  after  transformation)  is  required  for  tests  of  significance 
but  this  requirement  is  not  normally  a stringent  one. 


Further  discussion  will  be  facilitated  by  reference  to  Tables  2, 


3t  4,  and  5.  Table  2 gives,  purely  for  illustration,  the  linear  re- 
gression of  day  three  weights  of  mice  on  their  day  zero  weight  in  one 
laboratory  of  a twelve  laboratory  collaborative  trial.  Full  details 
of  the  experiment,  the  statistical  analysis  and  the  findings  are  re- 
ported elsewhere  (7),  (8)*  The  mouse  numbers  are  for  identification 
and  supply  no  information.  It  is  conceivable  that,  were  observational 
data  available  on  individual  mice,  such  information  could  suggest  ex- 
planations for  the  anomalous  weight  gain  behavior  of  specific  mice. 

The  initial  weights  are  shown  in  Column  2.  Column  3 gives  each  mea- 
sured three  day  weight  and  Column  4 gives  what  the  weight  of  each 
mouse  would  have  been  had  it,  been  exactly  what  the  regression  line 
computes  for  it.  Column  5 shows  how  large  the  discrepancy  between  the 
observed  weight  (Column  3)  and  calculated  weight  (Column  4)  is  and 
Column  6 shows  the  contribution  of  that  one  discrepancy  to  the  uncer- 
tainty in  every  deduction  from  the  data.  The  sum  of  Column  6 is 
25.3186.  From  Column  6 it  is  seen  that  mouse  2 and  mouse  5 each  con- 
tribute about  a third  and  together  two-thirds  of  the  total.  Mouse  5 
lost  weight  and  mouse  2 gained  well  above  expected . 

If  we  assume  that  the  day  three  weights  are  linearly  dependent  on 
the  initial  weights,  then  the  twenty  weight  observations  can  be  sum- 
marized by  three  quantities,  the  mean  initial  weight,  the  mean  final 
weight,  and  the  slope  of  the  line  relating  Initial  and  final  weight. 

Of  course,  the  observed  mean  and  slope  are  only  estimates  of  what 
would  be  observed  if  a very  large  study  were  to  be  carried  out.  How 
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good  an  estimate  is  measured  by  the  deviations  in  Column  5 of  Table  2. 
Provided  all  of  the  usual  assumptions  in  least  square  analysis  are 
valid,  this  measure  requires  only  the  sum  of  the  deviation  squares  of 
Column  6 and  their  number,  10.  Again  a study  of  the  adequacy  of  con- 
formance of  the  actual  assay  to  the  theoretical  ideal  involves  the  ob- 
served deviations,  or  residuals,  of  Column  5.  Failure  to  conform  to 
the  model  may  be  due  to  (a)  momentary  loss  or  gain  in  weight  due  to 
crowding,  fighting,  inequalities  in  food  sharing,  errors  in  weighing, 
and  so  on,  (b)  to  real  differences  in  rate  of  gain  by  individual  mice, 
(c)  to  heterogeneity  in  the  supply  of  mice,  (d)  to  non-normality  in 
the  distribution  of  weight  deviations  (Column  5,  Table  2)  and  (e)  to 
many  other  possible  factors. 

In  this  experiment  weights  of  each  of  ten  mice  were  measured  be- 
fore treatment  and  three  days  later  in  each  of  six  treatment  classes 
in  each  of  twelve  laboratories.  The  six  treatments  were  three  vac- 
cines in  two  replications.  Table  3 gives  the  initial,  observed  three- 
day,  and  adjusted  three-day  mean  weight,  and  the  slope  for  each  of  the 
six  vaccine-replication  combinations  for  the  same  laboratory  as  that 
of  Table  2.  It  will  be  noticed  that  the  mean  initial  and  day  three 
weights  of  Row  1 agree  with  the  means  of  Table  2.  The  adjusted  means 
of  Table  3 are  our  estimates  of  what  the  day  three  means  for  each  lot 
would  have  been  had  all  of  the  mice  weighed  15.18  grams  initially. 

The  slope  of  the  line  relating  the  three-day  and  zero-day  weights  is 
the  line  giving  the  best  fit  to  the  data  of  the  ten  mice  within  each 
treatment  class.  These  slopes  and  means  can  be  viewed  as  the  essen- 
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tial  inputs  to  the  covariance  analysis  of  the  one  laboratory’s  three- 
day  weights,  Table  4* 

It  is  possible  that  the  slope  relating  a variate  of  interest,  in 
this  case  the  three-day  weight,  and  an  unavoidable  nuisance  variate, 
in  this  case  the  initial  or  zero-day  weight,  may  be  known,  or  known 
so  accurately  that  it  is  considered  legitimate  to  treat  it  as  known. 
Doing  an  analysis  of  variance  on  weight  gains  is  equivalent  to  doing  a 
covariance  analysis  with  a slope  constrained  to  be  unity.  In  other  ap- 
plications a different  scaling  factor  might  be  more  appropriate.  The 
appropriateness  of  using  a known  regression  coefficient  and,  if  so, 
its  specification  is  not  discussed  in  Table  4.  It  is  provided  for  in 
the  covariance  program.  However,  the  complete  covariance  analysis  over 
the  twelve  laboratories  yielded  a regression  coefficient  indistinguish- 
able from  unity. 

There  is  a striking  variation  in  the  observed  slopes  for  the  six 
treatment  groups  for  this  laboratory  (Table  3).  The  mean  of  these 
slopes,  0.523,  is  only  half  the  value,  unity,  assumed  when  analysis  is 
carried  out  in  terms  of  weight  gains.  The  reality  of  this  variation 
in  slopes  is  examined  in  Tables  4 and  5. 

Further,  the  data  of  an  experiment  where  covariance  is  possible 
can  always  be  analyzed  to  yield  three  different  types  of  slope  esti- 
mates. Table  4 supplies  the  information  needed  to  check  the  adequacy 


of  each  of  these  methods  of  adjusting  three-day  weights  to  allow  for 
variation  in  initial  weights.  Depending  upon  which  procedure  is  cho- 
sen, a different  estimate  of  residual  variation  is  obtained.  These 
are  also  supplied  in  Table  4.  First,  all  of  the  data  can  be  entered 
into  a single  overall  regression,  yielding  a common  regression  esti- 
mate of  the  slope.  This  would  be  appropriate  if  the  treatment  classes, 
in  our  case  the  six  vaccine-replication  combinations,  were  without  ef- 
fect on  either  the  means  or  the  slopes  (section  14.7,  Snedecor  and 
Cochran,  1967).  Second,  the  data  of  each  treatment  class  on  its  own 
can  be  used  to  supply  a separate  slope  estimate  varying  for  each  class 
(Column  6,  Table  3).  A compromise,  and  the  method  normally  meant  when 
the  technique  of  analysis  of  covariance  is  used,  is  to  assume  that  the 
various  treatments  might  well  affect  the  mean  level  of  the  response  in 
each  class — it  is  usually  the  object  of  the  experiment  to  study  that 
question — but  to  assume  (subject  to  test)  that  the  treatments  (in  this 
study,  vaccines)  will  not  appreciably  affect  the  slope;  which  conse- 
quently will  remain  constant  over  the  classes.  For  each  of  these  three 
alternatives  a different  set  of  adjusted  means,  supposedly  removing  the 
effect  of  variation  in  the  nuisance  variate,  initial  weight,  will  re- 
sult. Of  course,  a fourth  alternative  is  simply  to  neglect  the  co- 
variate (or  to  show  that  it  is  without  effect). 

It  is  normally  assumed  in  a covariance  analysis  that  the  residual 
mean  squares  within  classes  vary  no  more  than  would  be  expected  by 
chance.  Hence  the  residual  mean  squares  of  the  six  vaccine-replica- 
tion classes,  each  with  eight  degrees  of  freedom,  are  pooled  in  line  1 
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of  Table  4 to  give  a single  estimate  of  1.4817  based  on  48  degrees  of 
freedom.  The  legitimacy  of  this  step  will  be  discussed  later.  Six 
of  the  lines  of  Table  4 are  concerned  with  slopes  (lines  2,  3,  4,  6, 

7,  8,  and  9).  Line  5 is  the  only  line  of  the  table  not  yet  listed. 

It  is  this  line  that  answers  the  question:  Was  there  a mean  effect 

on  mouse  weight  gain  by  either  the  vaccines  or  the  repetition  of  the 
trial? 

Even  if  the  adjusted  weights  are  significantly  different,  the 
slopes  for  each  different  vaccine  in  each  of  the  two  replications  may 
or  may  not  be  the  same.  This  is  tested  in  line  2 of  Table  4.  The 
non-significance  in  Column  7 implies  that  a given  change  in  three-day 
weight  for  a given  change  in  initial  weight  is  the  same  in  all  three 
vaccines  and  In  both  replications.  The  mean  square  of  Column  5,  line 
3,  1.5403,  hence  is  an  estimate  of  the  same  quantity  that  the  1.4817 
of  line  1 is.  Line  4 of  the  table  tests  whether  there  is  a regression 
line  with  non-zero  slope  relating  initial  and  three-day  weight  after 
allowing  for  a possible  variation  in  the  subclass  means;  the  fourth 
possibility  listed  above.  That  the  relation  tests  non-significant 
(Column  7,  line  4)  shows  that  the  variability  is  high  compared  to  the 
range  of  initial  weights  so  that  any  such  variation  does  not  clearly 
show  a dependence  between  three  and  zero-day  weights.  Of  course, 
there  was  in  fact  little  variation  in  the  initial  weights  themselves. 
Since  the  adjusted  weights  (line  5)  also  test  non-signif icantly  differ- 
ent, a single  regression  encompassing  all  60  readings  seems  justified. 
The  residual  mean  squares  from  this  regression  is  given  in  Column  5, 
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line  6,  and  the  reality  of  the  non-zero  slope  of  the  regression  in  line 
7.  Again,  such  a line  is  not  clearly  established. 

Lines  8 and  9 have  a different  meaning.  Each  of  the  six  treatment 
classes  (three  vaccines  in  two  replications)  yields  a mean  initial  weight 
x and  a mean  final  weight,  y.  The  six  such  pairs  of  readings  could  fall 
on  a line,  which  might  or  might  not  have  the  same  slope  as  the  average 
within-class  lines.  Whether  such  a line  meets  the  standard  test  of  signi 
finance  is  tested  in  line  9.  Whether  the  scatter  of  the  six  class  means 
depart  from  that  line  more  than  can  be  accounted  for  by  chance  is  tested 
in  line  8 of  Table  4.  Neither  tests  significant  in  this  one  laboratory. 
If  line  9 were  significant  and  line  8 not,  a presumption  would  be  shown 
that  the  adjusted  treatment  means  differed  only  because  the  average  ini- 
tial weight  of  the  mice  assigned  to  each  class  did  so,  and  hence  that  the 
nature  of  vaccine  treatment  was  ineffective.  For  this  conclusion  to  be 
appealing,  the  between-class  slope  would  have  to  agree  with  the  within- 
class  slope.  This  topic  is  discussed  more  fully  in  Section  14.7  of 
Snedecor  and  Cochran  (10) . 

A clearer  insight  into  the  meaning  of  the  lines  of  Table  4 can  be 
obtained  by  reference  to  Table  5.  Line  1 of  Table  5 gives  the  pooled 
corrected  sum  of  squares  and  products  in  all  six  treatment  classes  after 
adjusting  for  possible  treatment  effects  on  the  mean  values  of  the  ini- 
tial and  three  day  weights  (i.e,,  assuming  that  the  dependence  (regres- 
sion) of  three  day  weight  on  initial  weight  is  not  affected  by  the  treat- 
ments even  though  the  three  day  expected  weights  themselves  might  well 


be).  Notice  that  the  corrected  sum  of  squares  for  three  day  weights, 
y2=87.325*  is  the  sura  of  the  line  3 and  line  4 sums  of  squares  of  Table  4 
repeated  in  Columns  6 and  7 of  Table  5.  The  81.634  then  is  the  variabil- 
ity left  over  after  fitting  the  best  possible  common  slope  line  through 
the  mean  values  of  the  variates  in  each  of  the  six  classes  whereas  the 
5.691  is  the  benefit  obtained  by  doing  so.  Next,  assume  for  the  moment 
that  the  weights  of  all  mice  within  each  vaccine-replication  class  were 
identical.  Then  we  get  the  same  result  if  we  calculate  a regression  on 
the  six  group  means  as  if  we  use  all  the  data  in  a single  regression. 

The  necessary  corrected  sums  of  squares  and  products  are  shown  in  line  2 
of  Table  5.  Again  the  regression  sum  of  squares  and  the  residual  sum 
of  squares  agree  with  the  entries  in  Column  4 of  lines  8 and  9 of  Table  4 
But,  of  course,  the  initial  and  final  weights  for  every  mouse  within 
each  vaccine-replication  class  are  not  identical.  Ignoring  that  varia- 
tion and  using  all  the  data  in  a single  overall  regression  yields  the 
sums  of  squares  and  products  of  the  third  and  last  line  of  Table  5.  It 
is  easy  to  verify  that  the  entries  in  the  first  four  numerical  columns 
of  the  last  line  of  Table  5 are  the  sums  of  the  two  entries  just  above 
each. 


In  effect  then,  the  first  four  numerical  columns  of  Table  5 divide 
the  59  degrees  of  freedom  for  a single  regression  of  y on  x into  54  for 
a pooled  slope  regression  allowing  for  a different  mean  for  x and  y from 
subclass  to  subclass  and  5 degrees  of  freedom  for  a regression  of  the  yfs 
on  the  3?’s.  Now,  examine  the  columns  of  Table  5.  The  three  entries  in 
Columns  6 and  7,  Table  5,  each  appear  in  Table  4.  The  entries  in  Column 
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5,  Table  5,  are  the  sums  of  those  in  Columns  6 and  7 on  the  same  line. 
Hence,  the  corrected  sum  of  squares  for  y in  each  line  is  divided  into 
two  parts,  one  in  Column  7 representing  failure  of  a line  through  the 
overall  mean  (lines  2 and  3)  or  a line  through  each  subclass  mean  but 
with  a common  slope  (line  1)  to  account  for  all  of  the  variation  in  y. 

The  entries  in  Column  6 are  the  amounts  of  sums  of  squares  which  are 
accounted  for  in  each  case.  The  slopes  calculated  on  the  basis  of 
Columns  3 and  4 are  shown  in  Column  8.  If  treatment  class  were  without 
effect,  and  if  there  were  no  chance  variation,  the  slope  calculated  from 
each  of  the  three  lines  of  Table  5 would  agree  with  those  of  the  other 
two.  The  observed  wi thin-class  pooled  slope  is  0.625  and  the  slope  of 
the  regression  of  y on  x is  0.194.  Of  course  neither  is  close  to  unity, 
required  for  use  of  mean  gains.  Moreover,  they  do  not  test  significant- 
ly different  from  each  other. 

A distinction  should  be  noted  between  the  five  degrees  of  freedom 

in  line  2 of  Table  5 and  the  five  degrees  of  freedom  in  line  5 of  Table  4. 

The  former  dissects  the  observed  treatment  class  sum  of  squares  for  final 

weight  17.087  into  two  parts,  one,  0.353,  measuring  how  well  the  six 

class  means  for  final  weight  fall  on  a straight  line  and  the  other, 

16.734,  how  much  they  fail  to  do  so.  The  entry  in  line  5 of  Table  4, 

17.8023,  measures  the  overall  agreement  of  the  adjusted  class  mean  final 
weight,  that  is,  after  making  allowance  for  variations  in  initial  weights 
within  the  one  vaccine- laboratory-replication  grouping. 

The  detailed  exposition  of  the  covariance  analysis  above  is  only  to 
make  the  procedure  clear.  Such  a clear  understanding  is  required  to  ex- 
ploit the  output  of  the  covariance  program  which  is  the  principal  subject 
of  this  paper. 


Table  1 


Types  of  Statistical  Models 


Case 

Mean 

Deviation 

Sum  of  Squares 

Univariate 

x - (Zx^)/n 

di  = xi  - * 

lx*  - nx! 
1 

Analysis  of 

xi.~  ^xij ^ni 

Variance 

dij  * xij  - xi. 

>.E(x4  .“X  )z 
11  1. 

Regression 

y = (£yt)/n 

di  = y±  - y 

E (y  -a-bx  ) 2 

x =*  (Ex^) /n 

i i 

Analysis  of 
Covariance 

h: 

d±j  = yij  - yi. 

EZ(yij"Vbxij)2 

x±=  (Exij)/ni 

Notes : 


1.  Analysis  of  Variance  involves  K>1  classes, 

2.  Regression  involves  regressor  variable. 

3.  Covariance  involves  both. 

4.  For  further  discussion  see  text. 


TABLE  2 


Least  Square  Pit 
Dey  Three  on  Initial  Weight 
One  Vaccine  in  One  Laboratory 


Expected 

Deviation 

* ■ 

Day  3 

Deviation 

Mmiit 

1 

Initial  fiay  3 

VeiRht 

C3W*> 

UuM*d 

13 

18 

16,97 

1-.WJ4  . - 

1.0682 

2 

14,5 

20 

17,04 

■«.»}#? 

8,7609 

3 

16 

13 

16,82 

fl,8188' 

3.3080 

4 

3 

16 

14*3 

17 

14 

16,82 

17,04 

4#8  ■ 

0,0320 

9,2432 

6 

13 

16 

16,97 

•0,9664 

0.9340 

7 

15.3 

17 

16,89 

0,1074 

0.0113 

8 

16 

16.3 

16.82 

•0.3188 

0,1016 

9 

16 

18 

16,82 

1,1812 

1.3955 

10 

16 

17.3 

16,62 

0.6812 

0,4640 

Main 

13.43 

16.9 

16,9 

0,0000 

23.3186* 

* Sura 

• 

TABLE  3 

Group  Means 

for  One  La bo 

ratory 

Day  Thraa 

Weights  (grama) 

_ 

Moan  Weights 

hai 

3 

*•£ 

1 

Initial 

Final 

Adjusted 

giopa 

13.43 

16.9 

16,73 

•0.148 

7A 

1 

13,83 

16,8 

16,38 

2.762 

geline 

I 

13,30 

17.6 

17.33 

0.578 

3 

2 

14*80 

16,8 

17,04 

-0.364 

7A 

2 

13,00 

13,6 

13.91 

2.000 

Saline 

2 

14,70 

17.1 

17.33 

0,406 

Mean 

13,18 

10,82 

16,82 

0,323 

U17 


TABLE  4 


Analysis  of  Covariance* 

Day  Three  Weights  for  One  Laboratory 

Lint 

T 

1 
2 

3 

4 
3 
6 
7 

5 
9 

10 

* For  explanation,  s««  text. 


TABLE  5 

Regression  Variability* 

Day  Three  and  Initial  Weights  for  One  Laboratory 


8 lope 

d£ 

£ 

*2 

£ 

Reg 

Day 

Slope 

Pooled 

34 

14,33 

9.100 

87.325 

5.691 

81,634 

0.625 

Between 

3 

9,43 

1.823 

17.087 

0.352 

] 6 , 734 

0.194 

Overall 

39 

23*98 

10.925 

104.412 

4.976 

99,436 

0.455 

* For  explanation,  see  text* 


Effect 

df 

Sum  Squarea 

Mean  Square 

V 

Within 

48 

71,1231 

1.4817 

Bet,  Coeff* 

5 

10,5105 

2,1021 

1.4187 

CM  SI  Rat  88 

53 

81,6336 

1.5403 

CM  SI  Red  88 

1 

5.6914 

5.6914 

3,8411 

Adj , Hearn 

5 

17,8023 

3.5605 

2.4029, 

CM  Rg  Ret  SS 

58 

99.4359 

1.7144 

CM  Rf  Red  88 

1 

4. 9766 

4.9766 

3,3586 

Dev  Bet  Cl  SI 

A 

16.7344 

4,1836 

2.8235 

Bit  CUte  81 

1 

.3531 

.3531 

.2383 

Total 

39 

104.4125 

1.7697 

1*18 
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Note:  Recent  literature  on  covariance  is  extensive  but  scattered.  I 

know  of  no  comprehensive  treatment  from  a user's  point  of  view. 


O U V !\  i l\  U O O 

Li  Ab  I C P K ( ) GUAM 
DOC U Mb A 1 ATI  Ob 

i'he  HHM  (remark)  statements  are  located  at  the  end  or  this  program. 
If  the  number  or  Items  in  each  -data  set  is  fixed,  the  symoo.1  LI  , 
line  bo  must  oe  set  to  that  count.  If  uia  nunoer  or  entries  per 
set  varies  then  9E9  is  used  to  separate  eacn  set.  If  tno  number  of 
items  in  every  entry  (or  data  set)  ore  one  same  then  9n9  should  oe 
omitted.  Missing  data  are  listed  a bo.  if  data  is  read  as  obb 
then  tne  program  sKips  over  the  entry  and  omits  it  from  tne  count 
and  degrees  of  freedom.  I’he  syr.ool  /L 9 in  output  indicates  that 
that  quantity  was  not  calculated,  nr our am  data  entry  snould  begin 
in  line  223b  (tnis  deletes  pro.rin  example  data). 

Select  a code  number  from  the  four  tholes  at  the  and  or  the  program. 
The  ( + ) sign  is  for  "yes"  and  Uu  (-)  sign  is  ror  “nod  fixe  follow- 
ing list  will  assist  you  in  using  tn  taolus: 


S*WiC)L  CdUE  LI  id.: 

L At.uh 

1.  How  many  input  data  sets  are  there  ? a ju 

2*  How  many  items  are  there  in  eacn 

group  (if  fixed)?  mi  eo 

3.  How  will  the  data  be  ont;reu?  J2  j -d  1 you 

9 3=  I Converts  to  logs  (see  a.:,  6) 

4 . Is  the  tra  ns  forme  d or  u n t r a i is  for.  n e a 

input  data  to  be  printed?  cb  l-o  2uub 

b.  (a)  dill  the  printed  output  include 
tne  input  data?  (b)  * I i ) i In  ,• 
entire  output  oe  printers  or  only 

certain  parts?  ob  l-o  kudo 

6*  dhich  slope  will  form  the  u justed 

mean?  Lb  I -e  a 14b 


w()r  bo  * 

1.  Two  excellent  mutually  clariryin:  texts  are:  Jnedecor,  J.  u.  and 

Cochran,  a.  G. , "Statistical  ;otn  v is",  oth  edition  196/,  Chapter 
14,  and  Steele,  If.  G.  L.  and  Torrie,  J.  H.  , "Principles  and  Pro- 
cedures of  Statistics",  1900,  On ..ntur  Id. 

2.  The  program  computes  (virtually)  all  or  the  procedures  involving 
one  criterion  of  classiri cation  ..nd  one  regression  variable • 

3.  In  addition,  tne  program  permits  a cnoice  of  regression  coeffi- 
cient for  adjusted  means  including  a prior  assignment  and  a test 
of  significance  for  the  latter. 

4.  The  program  also  permits  a wide  choice  of  (a)  source  of  data, 

(h)  transformation  of  variates  and  (c)  print  out  format. 

d * If  the  classification  or  tne  da iu  sets  is  without  effect  on 

either  variate  tne  common  re  if  ess  ion  is  the  appropriate  one.  If 
not,  the  references  or  other  authorities  snould  oe  consulted. 

o.  while  the  program  has  been  v.  raiully  Checked,  ansolute  accuracy 
and  provision  ror  all  contin  ; on cj.es  cannot  oe  guaranteed. 

/.  suggestions  ror  corrections  ^ f program  or  documentation  errors  or 
infelicities  will  oe  appreci  ted. 
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C 0 V A R i A 1M  C u 
J Ab  f C An OCR j\M 
SYMBOL  INULA 

'-'LiNL  NJiitihU  UiiLRL  A I RSI 


SYMbOL 

aIHANINC 

UhT  1 NHD 
(),i  R1::AL) 

ud*:U  OR 
PR  I NTl:J 

A(J) 

Jncorrected  sum  of  X squared  ror 
Jth  data  set. 

odd 

93  b 

A 

Total  uncorrected  sum  or  X squared • 

9 3d 

970 

AS(I) 

Program  Headings* 

to 

bu 

3 ( J) 

uncorrected  sum  of  products. 

j6o 

65b 

3 

Total  uncorrected  sum  of  products. 

940 

97  b 

db 

Code  ror  selecting  printouts. 

00 

1 4u 

C(  J) 

Jncorrected  sum  of  Y squared. 

09  0 

060 

c 

total  uncorrected  sum  of  Y squared. 

9 4 0 

9dO 

Cl 

effective  error  mean  square  for 
regression.  X 

1 j30 

1 Bub 

cc 

Centering  constant  Tor  program 
headings. 

do 

do 

Cb 

Code  selecting  data  to  be  printed. 

0 U 

14b 

Cb 

Centering  constant  foT*  data  sec 
Headings. 

620 

32 1 

U(J) 

Sum  or  X for  Jth  data  set. 

o 9 0 

6 4b 

u 

Overu  11  ( total ) sfUa  or  X * 

900 

970 

D1 

directive  error  moan  square  for 
difference. 

1 ode 

1 dUb 

D2 

Prior  slope  (given). 

6 b 

1 u40 

ti  ( J ) 

Sum  or  f ror  Jth  dates  set. 

0 00 

6 40 

L 

Overall  (total)  sum  or  Y. 

9 5b 

97b 

61 

Reduction  sum  of  squares  ror  betv/oen 
classes  regression* 

1 abb 

1 56o 

*Symool  occurrences  in  quotes  are  disregarded. 

/3,h,o;  .. . 421 


oYMBOL  INDtiiX  continued. 


>YnBOL 

I'L  A i'«  I NO 

Li  Mu  aUrLuhd  y 
dOF  i J'Jl:  J 
Oii  iJL A 0 

iJHuiiiu  titl'd 

U O t-  '.J  Oil 
r H I MX  r.'  J 

t ( J ) 

Covariance  degrees  of  f: 

r o e u 

O.ii  . 

i py  j 

1420 

F 

Overall  (total)  wituin 
freedom. 

..itigi 

■JOS  of 

1 n-40 

1 460 

G(J) 

Sum  of  deviations  squar. 
data  set. 

. : v i r 

>r  Jth 

1 COD 

1420 

H ( J ) 

J . r * ror  corrected  sum  < 
and  products. 

of  s 

yivir  js 

./o  b 

1210 

HI 

Sun  or  U.t.  for  correct: 
squares  and  products. 

-'d  S 

: i ,i  of 

dd  D 

1 2 bb 

H9 

Set w sen  cue  f f i cion t s de 
freedom. 

; r o e 

0 i 

920 

1 bHO 

H$(i  ) 

Data  set  Headings , 

120 

32b 

Id) 

fra  ns  format  ion  symool  f.- 

>r  t 

• 

j/  D 

77  0 

J(i) 

franc format ion  symool  for  a 

• 

4 2 D 

4 o 01 

.<(1) 

Expected  value  or  r . 

/CD 

7lo 

K 

dumber  of  data  sots. 

DO 

1 lb 

L(  I ) 

Individual  deviation  sqi 

! r.  re- 

.  • 

/ 1 b 

73  b 

L 

Overall  (pooled)  sum  of 
within  lines. 

s'  ;l 

: r e s 

1 H-rD 

1460 

IJ 

affective  error  mean  s y 
Y only. 

i a re 

for 

1 D4J 

1 7bu 

L2 

Treatment  mean  square  f 

>r  y 

only. 

1 d4  d 

1 7 bo 

L5 

Code  ror  slope  In  ad jus. 
calculation. 

.0  i 

:oan 

dJ 

93  D 

M(J) 

deviation  mean  square. 

1 420 

1460 

M 

Residual  sum  of  squares 
slope • 

for 

common 

1 j 1 0 

1 59b 

k22 


SYhuOL  IdOLX  continued 


S Y i«ijJ  o L 

mdAOIIW 

LI  No  dU.  w 
Id  OF  I i'iiiO 
Os  d L AiJ 

end  j 

• duiiL  F liv'd  l 

ud.iL)  op 
Fill  WThiJ 

Mi 

dumoer  of  items  in  equal  data  sets. 

a a 

j da 

N(  J) 

Actual  count  (dL6s  noc  counted)  for 
Jtn  da  ta  a e t • 

J aO 

aid 

N 

Overall  (total)  count. 

92  a 

960 

0 

Sum. of  deviation  squared  for  common 
regression  residual  str  • or  squares. 

laid 

1640 

P(  J) 

Corrected  sum  of  X squared  r or  Jtn 

i i C-  * t o s v t . 

o o a 

6 7 u 

9 

overall  corrected  sum  of  X squared. 

o V 0 

9 1 d 

P7 

Juuioer  o r lines  or  program  nee  ding . 

ad 

7d 

Pcj 

Code  ror  converting  X to  lujs. 

ac 

■ 4 1 d 

Q ( J ) 

Corrected  within  sum  of  XY  for  Jtn  . 
lata  set. 

o ad 

6 7d 

0 

Overall  corrected  within  sum  or  XY. 

o9a 

9 1 a 

05 

Slope  ror  adjusted  means  (see  code  La) 

. 990 

1 02  d 

R C J ) 

dorr,  witnia  data  set  sum  of  i sqd. 

66  a 

9U0 

a 

Moo  leu  sun;  of  corrected  Y squares. 

■y  cu 

12  5a 

\u 

hum  or  deviation  .squared  from  oe tween 
classes  regression. 

1 ao  J 

i 560 

iU 

Mean  square  ror  deviation  rrora  net  ween 
classes  regression. 

' 1 Odd 

1670 

SC  JJ 

hi thin  class  slope  for  Jtn  data  set. 

o 0 a 

990 

s 

deduction  sun  of  squares  for  common 
slope. 

1 aud 

laid 

SI 

Value  ror  prior  slope  (see  Code  Lb) . 

o J 

1020 

:->k 

Code  ror  input  data  sources  ana 
t ran s ror mat ions. 

ad 

340 

b23 


SYMBOL  INDbX  continued 


LI  Me  N»oAfc>HR  riHHHb  FIRST 


SYMBOL 

MbANINu 

DbFI  bl- 
Od  Fa  Alt' 

DSHD  OR 
PH  I NT HD 

T$ 

Type  or  study. 

50 

150 

J(J) 

Mean  or  X ror  Jth  outu  set. 

645 

705 

U 

Corr.  total  sum  of  X squared. 

9/0 

1290 

US 

Overall  mean  square  for  treatment 
f or  X . 

1525 

1 5 50 

U9 

Overall  mean  square  to1'  residual 
for  X* 

I 520 

1725 

V(I) 

Deviation  or  Y from  expected  value 
for  Jtn  data  set. 

710 

735 

V 

Corrected  total  sue  oi  XY, 

975 

1 JI0 

W(J) 

Mean  of  Y for  Jtn  uatu  set. 

o40 

705 

fi 

Corrected  total  sun  oi  Y squares. 

930 

12  90 

XU) 

I tli  item  of  data  for  X, 

36  5 

4 30 

X$(I  ) 

Designation  for  I to  el  '..men  t or  data. 

o 1 5 

736 

YU) 

Ith  item  or  data  for  V. 

56  5 

5 7 0 

SC  J) 

Ad  just ed  m e a n o f V lor  Jtn  d u la  set. 

1 o30 

1 065 

7 b 9 

Quantity  not  calculated. 

6 55 

1 355 

dbd 

Missing  data. 

470 

1 350 

9b  9 

bnd  of  one  treatment  en-ts  set. 

4 7 5 

1345 

k2k 


\j<  w » V— l 1 l\ 

OK.  DATE  FILED*  07/31/74. 


LIST  5,245 


0 DIM  A ( 33 ) i B (3  b)  ,C(  35 ) , D( 3b ) , E ( 35 ) , F ( 3b)  , G( 35 ) , H ( 3b ) 

10  DIM  I ( 3 o ) , 3 ( So ) * K ( 35 ) f L ( 3d  ) , M ( 3 3 ) , N ( 3b ) , 0 ( 3b ) , P ( 35 ) 

15  DIM  Q ( 3 5 ) , R ( 3b ) , S ( 3 5 ) f T ( 3b  ) v <J  ( 3 3 ) , V I 3b ) , »» ( Jb ) t X C 3 5 ) 

20  01  M Y(  35)  ,Z(jq)  ,H$ ( 1 0)  ,XM  3b) 

25  I a Had  on 

30  OPbN  1,"CHRNa" 

35  OPEN  2 , "CnRNY" 

40  OPbN  3,  “HHUCriPN" 

45  DaI‘A  2 ,3,0,6 ,1  , 1 f 2 

bO  RlAd  P7  ,Kf  Pb  ,52  ,C5  ,85 , L5,  i'$ 

35  LbT  Mi  =1  Quo 

0 0 LlT  SIM 
b5  LcT  D2“ I 

70  PoR  1=1  TO  P/ 

/ 5 Rti AO  AMI) 

30  C2=0TL( AM I ) ) 

3b  P.7  In  TI  Abl  N i ( 30-C2/2 ) t A$  ( i ) 

90  N-XI  I 
9 b DATA 11 TH I AL" 

1 OU  JAi  A"  SPECIAL  TEST  PROM  dOChRAN  * CLI  FFORD  J.  MALOnEY  , PH.  0*  , Si  AT. 11 
105  JAi A“ COCHHAim  EXAMPLE " 

110  0/vTA"DRUG  A1',  "DRUG  B'V'dRUg  F" 
lib  F OR  1 = 1 TO  A 
120  READ  rISCI) 

1 2b  jmE’XT  I 
130  PRINT 
1 3b  PRINT 

140  IF  Bb>4  i'rlbN  3o5 

145  ON  Cb  GO  TO  1 50,1 70, 1 90 f2 1 0,230,250,270,290 

1 50  P jRI  NTTAB4  MG ; TAB  1 6 » H X ( I ) " ; i AB26  * " Y ( I ) 11 5 TA  339  * "DE  / 1 All  ON"  i TAb59  ? 

155  PRI NT" OEM. Duo. 11 

160  plU  NTTAb/bt  "JNTRANS.X(  I ) AND  V ( I ) " 

165  GO  TO  30b 

1 70  PRI NTXAB4  MG  » TAB  I 6; "X ( I ) " ? 1 AB26  ♦ " I ( I ) " U A339  ? "Ob  VI  All  ON"  i TAb59« 

175  PRI NT" DEV .SOO. " 

180  PRI NTT  Ad 1 4 * "UNTRANSFORMeU  X ( I ) , i C I ) . ( V ( I ) = X < I > SOD.  IN  PROGRAM) " 

18b  GO  TO  30 b 

190  PRI  NTTAB4  MG  * TAb I 6* "X ( I ) "» TAB26 ; " Y ( I ) " * i A3  39 ; "OtV I ATI  ON" J TAb59 i 
195  PRI NT" DEV *300. " 

200  PRI NTTAB 1 /i"JiURNS  X(I),TRNSO  YU)  (YU)=1(I)  SQD.  ) " 

20b  uU  TO  3Ub 

210  PRINTTAB4  MG  » TAB  1 6 ; " J ( I ) " 5TAB26; "Y( I ) " i TAB 39; "DEVIATION"  TTAB59 i 
215  P H I N T " Db  V . S 00 . " 

22u  PRI  NTTAb  1 4 i " J.N  TRANSFORMED  J(I)G(I).  (a(I)=LOU  J ( i ) IN  PROG.)" 

225  GO  TO  305 

230  PRI  N1TAB4  M ^ » TAb  I 6; "X ( I ) " M AB26 ; " Y ( 1 ) " i TAB  39 i "Db  VI  Ai  l ON"  ? T>vd59i 
23b  PRI  NT" Or.  V .GOD*  " 

240  PR1nTTAB2u» "TRANSFORMED  X(I).  (A(I)=J(I)  IN  LOGS)" 

245  GO  TO  305 
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1-101  D<iU,bUb 


b20  Lb  i'  ACJ)  ,6( J) ,C( J) ,D(J) ,E(J)=0 

52b  IF  S2=7  THEN  b3b 

D 30  00  TO  540 

b3b  RH^lNO  3 

540  H)R  1 = 1 TO  Ml 

545  ON  S2  00  TO  bbO,  o6b , 56b  , 550,  550 , 550  « 550  ,56b 

bbO  IF  Y(I)=8E6  THEN  635 

555  IF  Y(I)=9b9  THEN  640 

b60  00  TO  560 

b65  IF  I(I)=dh6  THEN  635 

b7 0 IF  1(1)  =969  THEN  640 

575  LET  Yf<I)=I(I) 

b80  LEi  A( J)=A( j )+X (1)^2 

b85  LET  B(J)=6(J)+X(I)*Y(I) 

b90  LET  C(J)=C( J)+Y(I)^2 

b95  LEE  D(J)=U<J)+X(I) 

oOO  LET  E ( J)=H<  J ) + Y ( I ) 

605  IF  S3  =6  THbN  630 
olO  IF  S2  = 7 THEn  630 
615  READ(2f0)  Xo ( I ) 

620  IF  S2=7  THEN  6 30 

62b  00  TO  635 

o3u  READ( 3,0 ) X$(I) 

6 3b  nEXT  I 

640  LET  N(J)=b< J)/N(J) 
o4b  LEi  U(J)=0( J)/N<J) 
o 50  LET  P ( J ) =A<  J ) - (D( J) ~2 )/N<  J) 
o 55  LET  Q(  J)=d(J)-(l>(  >/n(J) 

660  LEi  R(J)=0(J)-(E(  J)v'2)/iV(  J) 

665  LET  H(J)=N(J)-i 
O 70  IF  P(J) <0.0001  THEN  685 
o7 5 LEI  S( J)=0< J)/P(J) 

630  00  TO  690 

685  LET  S(J)=/b9 

690  IF  65 >4  THEN  830 

695  PRINTTAB32;nC0UNT=  “ ; N ( j ) 

/GO  FOR  1 = 1 TO  Ml 

/Ob  LET  K( I)=R(J)+3(J)*(X( I ) -U ( J )) 

/ 1 0 LET  V(I)=7(I)-K<I> 

/lb  LEi  L ( I ) = ( Y ( i ) -1<  ( I ))*2 

/20  ON  Cb  00  TO  /4b , 7/0 , 790 t 72b , 74b , / 5b , 3 1 0 , 7 5b 
/2b  IF  Y ( I ) =8 Ed  THEN  82b 
/30  IF  Y ( I ) =9E9  THEN  830 

/3b  PRINT  USING  76u,  X$  < I ) , J ( I ) , Y ( I ) , V ( I ) , L (I  > 

740  oO  TO  82b 

745  IF  Y ( I ) =8E8  THEN  825 

750  IF  Y ( I ) =9b9  THEN  8 30 

755  PRINT  USING  760,  X$ ( I ) , X ( I ) , Y ( I ) , V ( i ) , L( I ) 

7 60  b I bLD ( 4X , Ab ,4X,F6 , 3, 4X ,F6 . j ,3X,F I 6.8, 3X ,b 10. 8) 

765  00  TO  825 

770  IF  I ( I ) =8b8  THEN  825 

/7b  IF  I(I)=9b9  THEN  830 

780  PRINT  USING  760,  X$ ( I ) f X( I ) , I ( I ) , VC i ) ,L(I ) 

73b  GO  TO  82b 

790  IF  I ( I ) =8iio  THEN  82  5 

795  IF  I ( I )=9h9  THEN  830 

600  PRINT  USING  760,  X$  ( I ) ,X(  I)  , I ( I T'2  , V ( I ) ,L<  I ) 
805  00  TO  625 
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LIST  1080,1 320 


080  print 

088  PRINT 
090  PRINT 

098  PRINTTAB26;  “ i'OTAL  SUMS  ANL  MEArS" 

100  PRINT 

1 08  PR  I NTTABs; 11 SU  M-X'1  • TAB 2*; ; "Mb’ AN*  a"  ; 1aB3/  5 "SUM-Y" 1 TAB  53 » *# Mti  Aim— V 
110  P R I NTTABo 3i "CQuNT" 

118  PRINT  USING  1 120,U,D/N,E,b/N,N 

120  FIELJtlri  ,4X , F6 .2 , 1 X , 3C I X , FI  5. 4)  , 4X, F 3* ) 

128  ON  88  GO  TO  I I 3o  , 1 225,  I 22d  , 1 22b,  1 1 30, 1 225, 1 225,  I 225 
130  PRINT 

138  PRINTTA3I  / i "UnCQRRECTEu  Su.MS  02  SQUARES  AND  PROUUGTS" 

140  PRINT 

148  PR  IN  TTAB  I ♦ “SOURCE11 ; TAB  1 91" SUM  a-SQ"  * TAt*35;  "SUM  PROu*  *•  5 TAB 82  * 

180  PRINT"SUm  y'-SQ. " t TABS  3? 11  COUNT 11 

158  PRINT 

160  FOR  J=1  TO  R 

168  PRINT  USING  1 1 70 ,H$ ( J) , A< J ) ,B ( J ) ,C ( J ) ,N ( J ) 

170  FIELQ(AB,6A,3(iX,FT  5.4)  ,4X,F3,  ) 

175  NhXT  J 
180  PRINT 

1 8d  Pr IN TTAB  I 8;  "CORRECTED  BUMS  OF  SQUARES  ANO  PRODjCTs" 

190  PRINT 

198  PR  I N ITAd  1 i “0 . F . “ ? TAB  11  • " SU  M X-sQ . •< « TAB29  5 “SUM  XY'*iTAB45; 

200  PR  IN  T"SU^i  Y-SQ.  " 5TAB65  ♦ " SLOPt:" 

20 u FOR  J=l  TO  A 

210  PRINT  USING  1 2 1 5 , H ( J ) , 2 ( J ) , 0 ( J ) , R ( J ) , S ( J ) 

2 1 5 FT  ELD ( 1 X ,2  3.  , 4 ( 1 X ,F  16  . o ) ) 

220  Ns XT  J 

228  ON  s5  GO  10  1230,1  345,1230  J 34o, 1200,1348, 1230, 1348 
230  PRINT 

235  PrINTTAbbI  i11  ********  GOMMOa  ********" 

240  PRINT 

245  PR  I N TTAB  I ♦ “0.  F . " * TAB  1 I ? " Su  M X-SQ.  " i TAB29  i “SUM  XYM«!Ab45« 

250  PRINP'SUM  Y-SQ • “ ;TAB65 » "SLOPE" 

255  print  usirg  121 s*m1  ,p,u,r,o/p 
260  print 

265  PrINT  TAol  95"  ********  dE  Ta EEN  ^LASSES  ********** 

27 0 PRINT 

275  PR  IN  TTAB  l * " J.F • " *TAB1  1 *"SuM  X-SU.  **  t TAB29;  “SoM  XY"  i TAb45» 

2 So  PR  I N T"SUM  Y-SU.“5TAB65i  "SLOPE** 

2s 5 I 2 j-p <0.0001  Then  1 3oo 

290  PRINT  USING  I21u,(N-1 )-HI ,U-P,V-Q,m~R,(V-Q)/(U-P) 

295  GO  TO  1 31 0 

300  PRINT. USING  1 3Ud,(N-1 >-H1 ,U-P,v-Q,N-R 
306  F IELD ( 1 X ,Fs . , 3( 1 X , F 1 6 .6 ) ) 

310  PRINT 

315  PR  I NTTAB2 1 ; " ********  TOTAL  ********'* 

320  PRINT 


LIST  I 32  b f I bds 


1325  PaInTTABI  ?“L>.b.*‘;  TA31  1 ♦“SuM  i'AB29l*'SUU  XY»»*TAB4b« 

1 3 30  PAIiYP'SUM  Y-SO . 11  ? TAB6  5 * “SlOPi  “ 

133b  PAINT  USIimO  l 2 1 5 ,N-I  ,U  , V ,h  tV/J 
1340  PhINT 

1 345  PAI  N'fTABGui “ ***********x“ 

I 350  PHINT 

1355  PAI[m7vB23*',DbVlATION$  b’AOM  aBGaBSSI ON" 

1300  PAINT 

1 36  5 PAINP'bXPhAl  MbNI’S*'  ; TAB  i 7 ? “D.b  . " ; TAB20  i “SUM  JEV.SQ.  “ > i’Ab44  5 
13/0  PAIN  f 11  Mb  Aim  SOu  AAb“ * TABo2 b «...  “ 

13/5  PAINT 

1 3do  Boa  j=  i To  rv 

1355  lb  P(J)<u.bu01  iHLN  1 4u5 

l 390  LfcT  PC J)=n( J ) - 1 

) 395  Lbi  J(J)=0(J)-L(J)rt2/N(J)-Q(j)''2/P(J) 

UOo  GO  TO  1420 

i 405  LbT  G( J ) -C ( J ) -b ( J ) l/h  ( J ) 

1410  LbT  r(J)=h(J) 
i415  LbT  H9=H9-1 
i 42o  LbT  M( J)-G( J)/P( J) 

*425  NEXT  J 
1430  LbT  P,L=u 
:4  3b  POP  J=1  10 
i 440  LbT  r=F+r(J) 
i 445  LcT  L-L+GC  j ) 

•450  NbXT  J 
1455  POP  J- 1 i'O  j\ 

1400  PAINT  USING  1 465  f H$ ( J ) , P ( J ) , G( J ) , M( J ) f M( J ) *P/L 
l 465  F I BLOC  Ad  *dX  ,P  3.  , 3(  1 X , P 1 6 . o ) ) 

1470  NbXT  J 

1475  PAINITABI  bi" “ 

1480  PAINT 

; 485  PAlNTfAB2bf  “ANALYSIS  Or  GOVAi.  I ANCb" 

? 490  PAINT 

1 4 9b  P A I N T USInG  15 00  « b , L » L/b 
i 5 Ou  b I b LO  ( 11 A 1 i ri  1 N “ , 3X  , P 3.  , 2 t 1 ,< , b 1 0 • d ) ) 

1505  LbT  3=Q"2/P 
510  LbT  M-H-S 
' 515  LbT  0-A“V''2/U 
? 520  LbT  U9-P/nl 
; 525  LbT  U 8= ( j ™P ) / ( ( N - 1 ) -H 1 ) 

1530  LbT  0 1 »L/b  * ( 1 +ud/P ) 

’53b  Lb  i 01  = ( A”2  *02*'J+P*G2''V  ) /;  1 ! 
i 540  Lbi  Ll=A/ril 
i 54b  Lb  T L2=  ( ,*~U ) / ( ( n-  1 ) — H 1 ) 
i bbo  Ir  u-P<0.ouOl  THEN  157b 
i 555  Lbi  tl  = (V-0)*2/(U-P) 

1560  LbT  Al=N-A-b)  ffSUBTRnOTS  POOLbJ  nl THIN  GOA. S.SQS.bA  OOA.IOT.S.S. 

i 56  b GO  TO  15  /b 

1570  LbT  AI-n-A 

1575  Ir  H9<2  The A 1590 

1580  LbT  A2=A  1/(119-1  ) 

1 585  GO  TO  1 59b 
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COMPUTATION  OF  MOMENTS  OF  A LOG  RAYLEIGH 
DISTRIBUTED  RANDOM  VARIABLE 


William  L.  Shepherd 
Research  Projects  Office 
Instrumentation  Directorate 
White  Sands  Missile  Range,  New  Mexico 


ABSTRACT.  A log  Rayleigh  distributed  random  variable  is  considered  to 
be  one  for  which  there  are  numbers  a and  b,  a ^ 0,  b > 0,  such  that  its 
probability  density  function  is 

f(x,  a,  b)  = b(a<6gna)exp(ax  - bexpax)  , -®  < x < 00  . 

Formulas  for  recursive  computation  of  the  nth  moment  are  obtained  and  a 
convenient  table  of  constants  whose  use  facilitates  the  computation  is 
given. 

1.  INTRODUCTION.  In  [1]  and  [2],  a certain  translation  property  of  the 
log  Rayleigh  distribution  was  presented  and  used  to  obtain  a simple  formula 
for  computing  the  moments  from  a small  number  of  numerical  constants  and 
parameters  of  the  distribution.  In  this  paper,  we  derive  alternate  formulas 
to  those  in  [1]  and  [2],  whjch  offer  a somewhat  better  method  for  computing 
a second  set  of  numerical  constants  related  to  the  aforementioned  set. 

The  second  set  is  also  computed  recursively,  is  more  tractable  to  machine 
computation,  and  less  subject  to  accumulated  errors.  A brief  table  of 
the  altered  constants  is  Included.  The  derivation  of  the  required  formulas 
Is  self-contained  in  this  presentation  and  the  reader  need  only  refer  to 
[1]  and  [2]  for  background  information.  References  [4],  . . . , [6]  cover 
related  material  in  recent  engineering  literature;  [7]  is  a paper  on  extreme 
value  order  statistics  which  touches  on  a special  case  of  a log  Rayleigh 
distributed  random  variable. 

(1)  A random  variable,  X,  is  a Rayleigh  distributed  random  variable 
if  there  is  a constant  b > 0 such  that  the  probability  density  function 
(pdf)  for  X is 

2 

p(x,  b)  * 2bxe”bx  , for  0 x 

a 0 , for  x < 0 . 

(2)  Y is  a log  Rayleigh  distributed  random  variable  if  X is  a Rayleigh 
distributed  random  variable  and  there  is  a constant  a ^ 0 such  that 


It  can  be  shown  that  the  pdf  for  Y is 

f(x,  a,  b)  = b(a*gna)exp(ax  - bexpax)  , -®  < x < * 


(1.1) 


(3)  The  moments  are 


m (a,  b)  = /"m  xnf(x,  a,  b)dx  , n = 0,  1,  2 


(1.2) 


Formulas  for  computing  the  moments  are  obtained  in  Section  3.  Section  2 
presents  some  prerequisite  material  on  the  Riemann  zeta  function  and  on 
the  gamma  function  and  its  derivatives. 

2.  SOME  REQUIRED  FORMULAS  FOR  COMPUTING  THE  MOMENTS.  The  derivations  re- 
quire  a number  of  equations  from  [3J,  which  are  referred  to  in  the  notation 

of  that  book.  With  /°^(x)  and  r^°^(x)  referring  to  the  digamma  and  gamma 
functions,  Equation  6.3.1  of  [3]  is 

>)  ■ rPlOO  - . 


r^(x) 


or 


r^(x)  ■ ^°^(x)r^(x)  . 


(2.1) 


Differentiating  n-1  times,  we  obtain 

r(n)(x)  - "f  (J‘1H(n'1'j)(x)r(j)(x)  , n-1.  2.  . . ■ • 


J=0 

n-1 


r(n)0)  - l (?'1)/n"1‘j)(l)r(J)(l)  . 


j=o 


(2.2) 


(n)(l)  - (n-l)l  "f  (n-:T.jy!-  *(n'1_j)0)  -jf-  r(j)(l)  . 


from  which 


(-l)nr(n)(l)  (-1  )n 


in 


n-1 

I 

j=o 


1 

(n-1 -j )! 


(n-1 - j ) ) 


r^(l) 


. . (2.3 


Equations  (6,3.2)  and  (6.4.2)  of  [3]  are 

♦(0)(1)  = -y  . (2.4) 

*(j)(l)  » (-1)j+1jk(J+1)  . j = l,2.  ...  (2.5) 

We  will  use  the  notation 

C-i  = y (Euler's  constant) 

cn  - c(n)  , n = 2,  3,  . . . (Riemann  zeta  function). 
Substituting  (2.4)  and  (2.5)  into  (2.3)  yields,  after  manipulating. 


H)nr(n)(l)  _ 1 V,  H)jr(j)d)  n = 1 2 

n!  n Acn-j  j!  ’ n '’■* 


(2.6) 


With 


do  = 1 • 


d . H)Jr(j)(1)  , j . ! 2 

°j  j!  * 3 '•  c’ 


(2.7) 


(2.8) 


(2.6)  is 


n “ n Vj  dj  ’ 


n-1 


n =*  1 , 2,  . 


(2.9) 


Since  accurate  values  for  the  Riemann  zeta  function,  c(n),  are  available  in 
Table  23.3  of  [3],  Equations  (2.9)  and  (2.8)  permit  a recursive  computation 

of  r^(1).* 

Another  formula  for  r^CI),  needed  in  (3.3),  can  be  derived  from  Equation 
6.1.1  of  [3]  by  repeated  differentiation. 

r(Z)  = /“  t^e^dt 

r(n)(Z)  = j”  (,«nt)ntZ‘Vtdt 

r^n^(l)  = /”  (•tnt)ne’tdt  . (2.10) 

3.  FORMULAS  FOR  THE  MOMENTS,  mn^a*  ■ From  (1.1)  and  (1.2), 

mn(a,  b)  = /“w  xn(a4gna)exp[(ax  + £nb)  - exp( ax  + £nb)]dx  ....  (3.1) 

Letting  x * (£nt  - £nb)/a,  (3.1)  becomes 

_ /_  D __  ^ r°°  / tnt  - tnb  \n*„-t  1 J+, 

mn(a,  b)  - a JQ  ( 5 ) te  -jp  dt  . 

D 

With  0°  = 1, 

mn(a,  b)  * (-}-)n  " (J)(-£nb)n'j  /;  Unt)je_tdt  ....  (3.2) 

j 0 

♦From  (2.>i, (2.8)  and  the  Taylor  expansion,  about  x = 1 , of  r(x),  we  also 
have 

r(x)  = r(l)  - d-|(x  - 1)  + d2(x  - 1)Z  + . . . 

= 1 + d^l  - x)  + d2(l  - x)2  + . . . 
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, 0 < x < 2 . 


Combining  (2.10)  and  (3.2)  results  In 


mn(a,  b)  = n!  (-  -L)n  j (In b)"‘J  ....  (3.3) 


Using  (2.8),  this  becomes 


mn(a,  b)  - n!(-  -g-)n  J (£nb)n‘^dj  , n = 0,  1,  . 

J ^ 

In  particular. 


(3.4) 


m. 


(-1.  1)  - n!d„  = (-l)nr(n)(l)  . n*0,  1 


(3.5) 


4.  SUMMARY  AND  CONCLUSIONS  . With  5]  » r,  C„  = c(n),  (n  = 2,  3,  . . . ), 


dQ  = 1 , and 


= (-Dnr(n)(1) 


n 


iTT 


, n = 0,  1,  . . . , 


(4.1) 


we  have  derived  the  formulas 

df>  = ~rT  ?n-jdj  ’ n = 1 


, 2,  . . . , 


(4.2) 


1 \r\ 


mn(a.  b)  = n!  (-  — ) I -T?pj7r  («)  -Jdj 

J ^ 


n = 0,  1, 


(4.3) 


mn(-l,  1)  = n!dn  . 


(4.4) 


Formulas  (4.3)  and  (4.4)  were  presented  in  disguised  form  in  [2].  The  forms 
presented  above  offer  advantages  over  [1]  and  [2]  in  computing  m„(a,  b).  Also, 

(4.1)  and  (4.2)  together  offer  a way  to  compute  rv  '(1)  which  has  not  been 
found  in  the  literature. 

We  have  d]  = y s .5772156649015327.  For  j = 1,  2,  . . . ,50,  Table  I 
gives  numerical  results  in  computing  d.. 

J 


UU5 


TABLE  I 


j 

dj 

1 

.5772156649015327 

2 

.9890559953279726 

3 

.9074790760808863 

4 

.9817280868344002 

5 

.9819950689031452 

6 

.9931491146212762 

7 

.9960017604424315 

8 

.9981056937831289 

9 

.9990252676219549 

10 

.9995156560727774 

n 

.9997565975086013 

12 

.9998782713151333 

13 

.9999390642064443 

14 

.9999695177634821 

15 

.9999847526993770 

16 

.9999923744790732 

17 

.9999961865894733 

18 

.9999980930811309 

19 

.9999990464689111 

20 

.9999995232106057 

21 

.9999997615973444 

22 

.9999998807960191 

23 

.9999999403971250 

24 

.9999999701982676 

25 

.9999999850990354 

26 

.9999999925494849 

27 

.9999999962747315 

28 

.9999999981373621 

29 

.9999999990686798 

30 

.9999999995343395 

31 

.9999999997671696 

32 

.9999999998835847 

33 

.9999999999417923 

34 

.9999999999708961 

35 

.9999999999854481 

36 

.9999999999927240 

37 

.9999999999963620 

38 

.9999999999981810 

39 

.9999999999990905 

40 

.9999999999995452 

41 

.9999999999997726 

42 

.9999999999998863 

43 

.9999999999999431 

44 

.9999999999999715 

45 

.9999999999999857 

46 

.9999999999999928 

47 

.9999999999999964 

48 

.9999999999999982 

49 

.9999999999999991 

50 

.9999999999999995 

hk6 


The  numerical  results  presented  in  the  Table  support  the  conjecture  that 

{d  }°°  has  the  limit  1.  If  this  is  true,  it  is  easily  shown  that  the 
n n=o 

relative  error  in  either  of  the  formulas 
mn(-l,  1)  * nl 


(-1)nr(n)(l)  * r(n+l ) 

is  zero  at  n = 
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TYPE  II  ERROR  Of  THE  2X2  CONTINGENCY 
TABLE  CHI-SQUARE  STATISTIC 
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ABSTRACT . Around  the  turn  of  the  century,  K.  Pearson  introduced  the 
"mean  square  contingency" , 4,%  which  is  a measure  of  the  degree  of  associa- 
tion betvffien  2 discrete  variables.  When  applied  to  m x n contingency 
tables,  <j>2  is  identical  to  the  usual  chi-square  statistic  divided  by  the 
sample  size,  and  in  the  2x2  case,  ^was  identified  as  the  square  of  the 
correlation  coefficient , p . [2;282], 

For  this  note,  a certain  bivariate  Bernoulli  distribution  is  developed 
which  explicitly  involves  the  correlation  coefficient.  The  square  of  the 
maximum  likelihood  estimate  (MLE)  of  P is  found  to  be  equal  to  A This 
allows  the  use  of  known  large  sample  distributional  properties  of  MLE's  to 
develop  the  general  distribution  of  the  chi-square  statistic  and  also  to 
develop  the  type  II  error  of  the  contingency  table  test  of  independence  as 
a function  ofp  . Several  interesting  properties  of  the  bivariate  Bernoulli 
distribution  are  related. 


INTRODUCTION:  A BIVARIATE  BERNOULLI  DISTRIBUTION.  Let  X and  Y represent 
Bernoulli  random  variables.  That  is, 


PCX  = 1]  = Pr  PCX  - 0]  = l-pr  P[Y  - 1]  = p2,  and  P[Y  - 0]  - l-p2. 

If  X and  Y are  independent,  the  joint  probabilities  Trxy  = P[X  = x,  Y = y] 
are  given  by  p.,x  q1  “x  p?y  q21_y,  where  q]  = 1-p,  and  q2  - 1-p2- 


I f X and  Y are  not  independent,  then  at  least  one  of  the  cell 
probabilities  (s$y  tt.,)  must  differ  from  the  marginal  product  probability 
by  an  amount  (say  K.J  Thus  n...  = p-.p?  + K,  but  to  keep  the  marginal 
probabilities  unchanged  the  rimaining^cell  probabilities  must  be  altered 
so  that 


*10  ~ ^1^2  ” ^01  = ^1^2  ” ^ ^0  = ^1^2  + 

This  may  be  written  in  one  of  the  two  forms: 

(1)  txy  = (1  - 2x)(l  - 2y)  K + p/  q,1^  p/  q21‘y; 

"W  = 0 - 2x)(l  - 2$)  K + [t-x-Pl  (1  - 2x)][l  - y - p2  (1  - 2y)]. 

x,y  = 0,1 


1*1*9 


The  two  cases  are  given  in  tabular  form  below. 


Y = 0 Y 5 1 Y = 0 Y - 1 


x = 0 

qlq2 

] 

q1p2 

x = 0 

v2 + K 

q ip2  - K 

x = 1 

P Tq2 

.Plp2 

x - 1 

p-,q2  - K 

P,P2  + K 

► 

Independent  Non-Independent 

Since  E(XY)  = p,p?  + K,  E( X)  = p-, , and  E(Y)  - p«,  it  is  immediately 
apparent  that:  c 


(2)  Cov  (X,  Y)  ■ K 

Notice  that  since  Var  (x)  - p-,q and  Var  (Y)  * p2q2,  the  correlat1on 
coefficient, p , is: 

(3)  p = K //p1P2q1q2 

If  t rxy  are  given  by  (1)  and  yield  the  marginal  probabilities  p,  and  p2 
for  X and  Y,  then  by  formnmg  the  sum  t+ttqO  " ^lO’^Ol  * we 

(4)  K - tt00  - n10  tt0;  and 

(s)  p = (*•,-,  tt00  - ttq1  ir10)/ /(7Tn+7rio^^n+^oi^1Too+7Toi^Troo+7rio^ 


An  examination  of  the  illustration  above,  indicates  that  K may  never 
exceed  any  one  of  the  four  quantities  l-p,p2>  ^ _c> i ^2 ’ plq2  or  qlp2  and  may 
never  be  less  than  -p^.'  -qiq2»  "H-Pi^  or  -0-qiP2)  since  alrcell 

>y  (rand  1.  Then  p^  = - p^2  ^ l-p-jP^ 

etc. , so  that 


probabilities  are  bounded 
^lp2  = P1  "p1^2  < ^ 


(6)  -min(p1P2>  q^)  < K£min(p1q2,  q1P2). 

Now  when  p1  > p2  then  0 < p1  - p2  = P1  - P^  **  (P2  “ P1P2)  = P1(l2  - P2b-| 

so  that  min(p-|q2i  q-|P2)  = q-|P2*  Also,  when  P]  < q2»  then  0 < P-|  - q2  - 

P i p2  " (1-P2-P1+P1P2)  = P\?2  - q-,q2  so  that  mintp-jPg,  q^)  = q^. 


Then 


. pr  > p2 


K < 


(7) 


Pi  < P2 


K > 


* P]  > ^2 


The  joint  moment  generating  function  (MGF)  of  X and  Y is  easily  seen 
to  be 


Mx,y(9T  V = [l-P1(l-eel)][l-P20-e’t)]  + K(l-6HI)(l-e^). 

It  follows  directly  that  if  X.  and  Y.  are  distributed  as  X and  Y 
respectively,  if  W = X,  + X?  ...  X , and  X = Y,  + Y-  . ....  Y , then 

the  joint  MGF  of  (M-npi)//~ni)1q1  and  ?Z-np„) /nipk  ig  (apart  fPom  terms 
of  order  1/n  in  the  exponent)  L c c 

exp  [l/2(ei  + 2 el02  +e2^  * 

and  therefore  W and  Z are  approximately  jointly  normally  distributed 
with  mean  np-j  and  np2  and  correlation  coefficient  p . 

MAXIMUM  LIKELIHOOD  ESTIMATES  OF  pr  p2  and  K.  The  likelihood  equation 
L ^=1  Vi  is 

(8)  L =nni=1  [(l-2xi)(l-Zy1)K+p1x1q11-x1p2y1q21"yi]. 


62. 


>el 


e2. 


Differentiation  of  In  L with  respect  to  p,,  p?  and  K yields  the  following 
set  of  equations:  1 L 


n00 

n10 

- 0 

*oo 

*10 

n00 

n01 

= 0 

*00 

*01 

o 

o 

c 

nll 

= 0 

*00 

*11 

b51 


where  n represents  the  number  of  observations  for  which  X = x,  Y = y. 
xy 

The  solution  of  (9)  is: 


(10)  p1  » x 


K = 


(nnnoo 


A 

p2  -y 

nionoi)/n 


If  equations  (10)  are  consistent,  then  from  a result  of  Chanda  [1] 
they  are  the  unique  MLE  of  p, , p9  and  K.  We  will  demonstrate  consistency. 
Let  1 1 


- 


1.  Xi=x,  Y.=y 
0,  otherwise 


Then  E tr^n,^)  = E[Li * n(n-l ) (p^P2  + K) Cq^q^  + K)  and 
^lO'W  = ^(n- 1 ) (P-jq2  “ K) (9-]P2  " K),  Thus  Etn^n^  - n0]nio^/n2  = 

A 

(n-l)K/n  so  K is  asymptoti cally  unbiased.  Similarly,  (the  algebra  is  inordate) 


V(lO  = KZ  (n-l)(2-n)/n3  + K(q1-p1)(q2-p2)(n=l)2/n3  + P1P2q1q2(n-l)/n2 , 


and  neglecting  terms  of  order  greater  than  1/n, 

(11)  V(K)  * [-K2  + K(q1  - p1)(q2  - p2)  + P1P2q-,q2]/n. 

A 

Therefore;  J<  is  a consistent  estimate  of  K.  It  follows  from  [1]  also  that 
p, , $2  an<i  K possess  a multivariate  normal  distribution.  We  may  then, 
for  large  samples,  assume  that  ft  is  normally  di s tribute d wi th  mean  K and 
variance  V(x)  given  by  (11).  More  precisely ,/n (1C - K)//nV(x)  is 
asymptotically  N(0,1). 

THE  ASYMPTOTIC  DISTRIBUTION  OF  P . To  obtain  the  MLE  for  P we  note 
that  w'p^q^  is  a one  to  one  transformation  from  K to  p since  the 

Jacobian  of  the  transformation  u-^e-))  * p-j,  u2  (e^)  * p^ , and  u^Ce^)  = 

K/ ypip2q-|q2  i^/e^  e2  ( 1 -©1) ( 1 -©2 ^ > Therefore  from  (9)  we  have. 


(12) 


nlln00  " n01n10 

nV 


That  (12)  converges  stochastically  top  follows  from  a theorem  of  Slutsky 
[2;255]  (since  converges  stochastically  to  /p^p^q^q^)  and  from 

the  result  of  the  last  section. 


A 

It  is  interesting  to  note  that  p may  be  written 


(I3)  ^ nlln00  “ n01n10 

p “ 

^(nn+nio)(nio+noo^noo+noi^noi+ni^ 


» 


and  that  [p  | is  the  well  known  phi  coefficient  which  is  used  to  estimate 
the  "degree  of  association"  in  2 x 2 contingency  tables  [2], 


To  obtain  the  distribution  of  P note  first  that  from  the  result  of 
the  preceding  section,  approximately. 


(14) 


yn 


nlln00  ' n10n01 

n2>lW2 


p 


( n V (K)  \ 

(s0,"PtP2V2  j ' 


Since  /To  -x)  y (1-y  )//  plp2qlq2  cor,ver9es  stochastically  to  1,  then 
combining  this  with  (14)  it  follows  from  a result  of  Cramer  [2;  254]  that, 
asymptotical  ly , 

yrf(^-'p)  (0,  n V ( K) /p-j p2q  1q^) 

A 

Thus  for  large  samples,  p is  approximately  normally  distributed  with 
mean  p"  and  variance, 

(15)  v (p  ) = Cl- p 2 +p  (q1-p1)(q2-P2)/ 


THE  LARGE  SAMPLE  DISTRIBUTION  OF  THE  2x2  STATISTIC.  If  we  denote  the 
2x2  chi-square  statistic  by  ’ t*ien  from  ^)  we  *iave 

(16) 


* 2 2 
n p - x 


2x2. 


From  (14)  and  (16)  it  follows  that  approximately  for  large  samples. 


(I7)  Xs2x2  -V  [n  V ( P )].  X'8,  (P2/V  (p*». 

JcL 

where  X v (0)  represents  a non-central  chi-square  random  variable  with  v 
degrees  of  freedom  and  non-centrality  parameter  0. 


*+53 


APPROXIMATION  FOR  THE  TYPE  II  ERROR  OF  THE  2x2  TEST.  The  type  II 
erroFof  th*  2x2  contingency  taBle  test  may  be  computed  for  large 
samples  from  (15)  and  (17),  but  It  is  easier  to  use  the  normal  distribution 
of  p . For  example,  the  .05  critical  value  for  the  chi -squared  test 
with  1 degree  of  freedom  is  approximately  3.841.  Then,  for  moderately 
large  samples,  we  may  write 

6 = P [42  < 3.841  |P  / 0],  or 

6 « P [fn  P < <T84i]  - P [4”n  P < 1 3.841]. 

Notice  that  if  p,  «■  1/2,  p-  = 1/2  then  V ( p ) = (l-P'2)/n.  This  is  an 
interesting  special  case  because  the  variance  is  not  dependent  on  p,  or 
p2  and  because  K has  the  largest  range  of  possible  values  (see  (6)  and  (7)) 

For  this  case  we  have  - 1/4  < K < 1/4,  -1  <p  <1,  and 

6 = P[N(0,  1)  < (<  3.84l‘-PfrT/\fl-p2)]  - 

P[N(0,1)  < (-  3.841  - p frO/Vd-P  7)}. 

In  order  to  indicate  the  amount  of  error  associated  with  the 
approximation  given  in  this  section,  the  following  tables  compare  the 
actual  type  II  error  obtained  from  (1)  and 

/ n \ n00  n01  n10  n11 

^nOo>noi>nio>nll)  17  00  1I01  "10  ’’ll  ’ 

with  the  approximate  values  obtained  from  (18)  and  (15)  for  several 
cases.  Notice  that  by  changing  the  values  of  p,  and  p„  to  1-p,  and 
l-Pp,  respectively  or  by  changing  only  one  of  pi  or  p„^along  with  the 
sign  of  p as  given  below,  the  tables  may  be  used  for  4 separate  cases 
each  (the  symmetric  case  p-,  - p0  = 1/2  excepted).  This  may  be  seen 
from  (15)  and  (7).  1 1 


PROBABILITY 

OF 

ACCEPTING 

Ho:  P 

= 0 

COMPARISON  OF  APPROXIMATE  VALUES 

. WITH 

EXACT  VALUES 

^ 

P2 

= 1/2; 

n 

= 25 

a = 

.1 

a = 

.05 

a = 

.01 

1*1 

ipi 

APPROX 

EXACT 

APPROX 

EXACT 

APPROX 

EXACT 

0 

0 

.900 

.884 

.950 

.948 

.990 

.989 

-p- 
v n 

.025 

.1 

,860 

.841 

.922 

.920 

.984 

.979 

V 1 

\_n 

.050 

.2 

.741 

.717 

.835 

.831 

.946 

.942 

.075 

.3 

.560 

.531 

.685 

.678 

.870 

.860 

.100 

.4 

.349 

.327 

.483 

.474 

.735 

.714 

.125 

.5 

.162 

.156 

.266 

.266 

.535 

.508 

.150 

.6 

.045 

.051 

.097 

.108 

.298 

.283 

.175 

.7 

.005 

.010 

.016 

.026 

.098 

.106 

.200 

.8 

.000 

.001 

.000 

.003 

.009 

.019 

.225 

.9 

.000 

.000 

.000 

.000 

.000 

.001 

PROBABILITY  OF  ACCEPTING  Ho: p =0 
COMPARISON  OF  APPROXIMATE  VALUES  WITH  EXACT  VALUES 


|K| 

M 

a = . 

APPROX 

P,  = 1/2; 

1 

EXACT 

P2  = 1/2; 

a = 

APPROX 

n = 50 
.05 

EXACT 

a = .01 

APPROX  EXACT 

0 

0 

.900 

.890 

.950 

.943 

.990 

.990 

.025 

.1 

.818 

.804 

.892 

.881 

.969 

.971 

.050 

.2 

.592 

.573 

.711 

.692 

.882 

.885 

.075 

.3 

.309 

.295 

.433 

.413 

.683 

.686 

.100 

.4 

.098 

.097 

.171 

.165 

.391 

.397 

.125 

.5 

.015 

.017 

.034 

.038 

.134 

.145 

.150 

.6 

.001 

.001 

.002 

.004 

.019 

.025 

.175 

.7 

.000 

.000 

.000 

.000 

.000 

.002 

.200 

.8 

.000 

.000 

.000 

.000 

.000 

.000 

.225 

.9 

.000 

.000 

.000 

.000 

.000 

.000 

PROBABILITY  OF  ACCEPTING  Ho:  P = 0 


COMPARISON  OF  APPROXIMATE  VALUES  WITH  EXACT  VALUES 
p1  = 3/4;  p2  = 3/4; 

n = 25 

P-j  = 1/4;  p2  = 1/4; 
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♦Change  the  sign  of  p to  obtain  the  table  of  values  for: 


DANIEL  AWARDED  THE  1974  SAMUEL  S.  WILKS  MEMORIAL  MEDAL 
Introductory  Remarks  Made  by  Frank  E.  Grubbs,  Conference  Chairman 


The  Twentieth  Design  of  Experiments  Conference  in  Army  Research, 
Development  and  Testing  marks  a very  significant  milestone  for  Statistical 
Methods  in  the  Army.  As  we  look  back  over  all  the  conferences,  it  is 
easy  to  see  that  we  owe  the  greatest  of  debts  to  Sam  Wilks  for  his 
vision  in  getting  Army  statisticians  together  on  a yearly  basis  for  the 
common  good  of  all.  Indeed,  we  have  benefited  much  from  our  twenty 
conferences,  and  I don’t  see  now  how  we  could  have  gone  through  the 
years  without  them.  Do  you?  Every  time,  I have  gone  home  with  a new 
view  of  statistics  in  the  Army,  the  papers  by  our  University  friends 
have  kept  us  up  to  date  and  provided  good  summaries  of  timely  accomplish- 
ments, and  we  have  been  motivated  to  either  attack  old  problems  with 
new  vigor  or  to  address  new  pressing  problems.  We  have  not  stuck  to 
the  title,  "Design  of  Experiments,"  in  all  detail,  but  that  is  good.  The 
field  changes  fast  and  we  must  always  move  on  to  new  things  or  areas, 
for  example,  reliability,  to  mention  one.  I could  go  on  and  on  concerning 
the  good  these  conferences  have  done,  and  Churchill  Eisenhart  has 
covered  much  of  that  so  well  anyway,  so  I’ll  stop  on  this  point  now.  But 
I must  mention  that  the  success  of  these  conferences  would  not  have 
been  so  great  were  it  not  for  our  most  dedicated  friend,  Francis  Dressel, 
who  as  we  all  know  deserves  a vote  of  thanks  at  this  time  for  his 
effective,  continuing  contributions. 

Now  I would  like  to  not  introduce  but  name  people  at  the  head  table 
that  we  know  so  well  and  have  enjoyed  association  with  over  the  years 
in  our  design  conferences. 

We  now  turn  to  the  Samuel  S.  Wilks  Memorial  Medal. 

The  Samuel  S.  Wilks  Memorial  Medal  Award,  initiated  jointly  in  1964 
by  the  U.  S*  Army  and  the  American  Statistical  Association,  is 
administered  by  the  American  Statistical  Association,  a non-profit, 
educational  and  scientific  society  founded  in  1839.  The  Wilks  Award 
is  given  each  year  to  a statistician  and  is  based  primarily  on  his 
contributions  to  the  advancement  of  scientific  or  technical  knowledge 
in  Army  statistics,  ingenious  application  of  such  knowledge,  or  successful 
activity  in  the  fostering  of  cooperative  scientific  matters  which 
coincidentally  benefit  the  Army,  the  Department  of  Defense,  the  U.  S, 
Government , and  our  country  generally . 
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The  Award  consists  of  a medal,  with  a profile  of  Professor  Wilks  and 
the  name  of  the  Award  on  one  side,  the  seal  of  the  American  Statistical 
Association  and  name  of  the  recipient  on  the  reverse,  and  a citation 
and  honorarium  related  to  the  magnitude  of  the  Award  funds.  The  annual 
Army  Design  of  Experiments  Conferences,  at  which  the  Award  is  given 
each  year,  are  sponsored  by  the  Army  Mathematics  Steering  Committee  on 
behalf  of  the  Office  of  the  Chief  of  Research,  Development  and  Acquisition, 
Department  of  the  Army. 

Previous  recipients  of  the  Samuel  S.  Wilks  Memorial  Medal  include  John  W. 
Tukey  of  Princeton  University  (1965),  Major  General  Leslie  E.  Simon 
retired  (1966) , William  G.  Cochran  of  Harvard  University  (1967)  , Jerzy 
Neyman  of  the  University  of  California,  Berkely  (1968),  Jack  Youden 
(1969)  retired  from  the  National  Bureau  of  Standards  and  deceased, 

George  W,  Snedecor  (1970)  retired  from  Iowa  State  University,  Harold 
Dodge  (1971)  retired  from  Bell  Telephone  Laboratories,  G.E.P.  Box 
(1972),  University  of  Wisconsin,  and  H.  0.  Hartley  (1973),  Texas  A.  and 
M.  University. 

With  the  approval  of  President  Jerome  Cornfield  of  the  American 
Statistical  Association,  the  Samuel  S.  Wilks  Memorial  Medal  Committee 
consisted  of  Bob  Bechhofer  (Cornell  University) , George  Box  (University 
of  Wisconsin) , Joe  Cameron  (National  Bureau  of  Standards) , Fred 
Frishman  (Internal  Revenue  Service) , Oscar  Kempthorne  (Iowa  State 
University),  Albert  Madansky  (Great  Neck,  N.  Y.  ),  Bill  Pabst  (Retired 
from  the  Navy),  Les  Simon  (Major  General,  Retired)  and  of  course  the 
esteemed  chairman,  Stu  Hunter  (Princeton  University).  What  an  array 
of  intelligentsia  to  select  the  1974  Wilks  Medalist! 

The  1974  Wilks  Medalist  was  born  in  1904  in  Williamsport,  Pennsylvania. 

He  secured  a Bachelor  of  Science  degree  and  a Masters  degree  in  Chemical 
Engineering  from  the  Massachusetts  Institute  of  Technology  in  1925 
and  1926,  He  later  studied  briefly  at  the  University  of  Berlin  and  at 
Harvard.  After  a short  stint  as  a teacher,  he  became  a research  associate 
in  the  Evaluation  of  School  Broadcasts  at  Ohio  State  University  and 
later  a research  associate  with  the  Princeton  Office  of  radio  Research. 

It  was  during  this  period  that  he  began  his  acquaintance  with  S.  S. 

Wilks  and  the  statistical  fraternity  at  Princeton.  During  World  War  II, 
he  was  employed  as  a statistician  with  the  Manhattan  Project  in  Oak  Ridge, 
Tennessee.  From  1947  onwards  he  has  been  a private  consultant  to  the 
food,  steel,  chemical  and  pharmaceutical  industries.  For  a period  he 
was  also  a consultant  to  the  U,  S.  Army  at  the  Signal  Research  Laboratories 
at  Fort  Monmouth,  New  Jersey.  He  has  also  been  a consultant  for  Consumers’ 
Union,  helping  with  organizing,  planning  and  analysis  of  its  comparative 
experiments.  He  is  a member  of  the  Cancer  Clinical  Investigations  Review 
Committee  of  the  National  Cancer  Institute  and  has  served  as  a consultant 
the  the  Office  of  Air  Pollution.  He  was  Chairman  of  the  Gordon  Conference 
on  Statistics  in  Chemistry  and  Chemical  Engineering  in  1954,  and 
collaborated  with  W.  J.  Youden  and  S.  Lee  Crump  in  the  early  establishment 


of  these  annual  conferences.  He  was  Chairman  of  the  Section  of  Physical 
and  Engineering  Sciences  of  the  ASA  in  1959.  He  has  published  widely 
in  the  JASA,  Biometrics  and  Technometrics,  has  been  a contributor  to  the 
Berkeley  Symposia  and  to  journals  in  the  engineering  professions.  He 
co-Authored  with  F.  S.  Wood  the  text  "Fitting  Equations  to  Data"  (1971) 
and  he  is  currently  completing  a manuscript  on  the  design  and  analysis 
of  industrial  experiments.  This  latest  manuscript,  although  still 
unpublished,  is  already  recognized  for  its  novel  contributions  to  the 
organization  and  interpretation  of  factorial  experiments.  He  was  the 
R.  A.  Fisher  Memorial  lecturer  to  the  joint  meetings  of  the  ASA  and  IMS 
in  1971,  and  the  W.  J.  Youden  memorial  lecturer  for  the  American  Society 
for  Quality  Control  in  1974.  He  was  one  of  the  original  associate 
editors  of  Technometrics.  He  is  a fellow  of  the  American  Statistical 
Association  and  of  the  Institute  of  Mathematical  Statistics.  He  has 
attended  and  contributed  to  many  of  these  Army  Design  of  Experiments 
Conferences.  He  is  well  known  to  us  for  his  unique  talks  and  commentaries, 
and  listerners  are  assured  of  an  entertaining  as  well  as  education 
experience  whenever  he  speaks  at  a meeting.  He  is  most  adept  at 
combining  theory  to  the  positive  solution  of  practical  problems,  and  is 
perhaps  the  Nation's  most  outstanding  statistical  consultant  to  the 
engineering  and  industrial  sciences. 

Now,  the  winner  of  the  1974  Wilks  Medalist  will  be  announced  by  Jerry 
Cornfield . 

CUTHBERT  DANIEL  RECEIVES  THE  1974  SAMUEL  S.  WILKS  MEMORIAL  MEDAL 

The  Presentation  of  the  Award  Made  by  Jerome  Cornfield, 

President  of  the  American  Statistical  Association 

The  following  citation  was  read: 

"To  Cuthbert  Daniel  in  recognition  of  his  outstanding  contributions 
to  the  applications  of  statistics  in  the  sciences,  for  his 
researches  in  the  novel  uses  and  interpretations  of  factorial 
designs,  and  for  his  vigor  in  stimulating  rewarding  colloguy  at 
statistical  meetings." 

ACCEPTANCE  REMARKS  OF  CUTHBERT  DANIEL  ON  RECEIVING 
THE  SAMUEL  S.  WILKS  MEMORIAL  MEDAL  FOR  1974 

Mrs.  Wilks,  President  Cornfield  and  fellow  Statisticians:  - 

Like  many  others,  I was  helped  by  Sam  Wilks  from  the  moment  of  my 
first  meeting  him.  He  arranged  for  my  first  real  contact  with 
statistics  in  1944  by  asking  a young  member  of  his  gorup  at  Princeton 
(A. Mood)  to  tutor  me.  Although  younger  than  I,  Sam  was  my  Dutch  Uncle 
and  advisor  many  times  over  the  next  25  years. 
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Wilks  was  the  first  of  a long  line  of  statisticians  who  have  given  me 
aid.  I think  that  for  my  case  the  award  should  have  added  to  its 
present  inscription , the  words  "with  a little  bit  of  help  from  his 
friends,"  A great  deal,  actually.  Allan  Birnbaum  (who  both  named  and 
linearized  the  half-normal  plot),  Henry  Scheffe'  (whose  memoranda  over 
22  years  I have  just  found  in  four  file  folders  all  labelled  Scheffe', 
Current),  Gus  Haggstrom  (who  has  worked  manfully  to  keep  me  more  honest 
than  I had  planned  to  be) , and  Fred  Wood  (who  is  a difficult  colleague 
because  of  his  insistence  on  getting  everything  right) , are  only  four  of 
the  most  helpful  of  my  colleagues. 

I thank  you,  then,  and  accept  with  full  recognition  of  my  past  and 
continuing  dependence  on  my  statistical  friends. 


1+62 


